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Abstract 

Small  Unmanned  Aerial  Systems  (SUAS)  have  been  utilized  by  the  military, 
geological  researchers,  and  first  responders,  to  provide  infonnation  about  the 
environment  in  real  time.  Hyperspectral  Imagery  (HSI)  provides  high  resolution  data  in 
the  spatial  and  spectral  dimension;  all  objects,  including  skin  have  unique  spectral 
signatures.  However,  little  research  has  been  done  to  integrate  HSI  into  SUAS  due  to 
their  cost  and  form  factor.  Multispectral  Imagery  (MSI)  has  proven  capable  of  dismount 
detection  with  several  distinct  wavelengths.  This  research  proposes  a  spectral  imaging 
system  that  can  detect  dismounts  on  SUAS.  Also,  factors  that  pertain  to  accurate 
dismount  detection  with  an  SUAS  are  explored.  Dismount  skin  detection  from  an  aerial 
platform  also  has  an  inherent  difficulty  compared  to  ground-based  platforms.  Computer 
vision  registration,  stereo  camera  calibration,  and  geolocation  from  autopilot  telemetry 
are  utilized  to  design  a  dismount  detection  platfonn  with  the  Systems  Engineering 
methodology.  An  average  5.1 12%  difference  in  ROC  AUC  values  that  compared  a  line 
scan  spectral  imager  to  the  prototype  area  scan  imager  was  recorded.  Results  indicated 
that  an  SUAS-based  Spectral  Imagers  are  capable  tools  in  dismount  detection  protocols. 
Deficiencies  associated  with  the  test  expedient  prototype  are  discussed  and 
recommendations  for  further  improvements  are  provided. 
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INTEGRATION,  TESTING,  AND  ANALYSIS  OF  SMALL  UNMANNED  AERIAL 

SYSTEMS  FOR  SKIN  DETECTION 


I 


1.  Introduction 

1.1  General  Issue 

ntegrating  Multispectral  Imaging  (MSI)  on  a  Small  Unmanned  Aerial  System 
.(SUAS)  may  present  a  valuable  tool  for  dismount  detection.  SUAS  offer 
versatile  platforms  for  Military,  police,  or  Search  and  Rescue  (SAR)  operators.  The  U.S. 
Geological  Survey,  the  U.S.  Air  Force,  and  others  have  demonstrated  the  utility  of  SUAS 
in  natural  disasters  such  as  the  Fukushima  Daiichi  meltdown  to  take  photographs  at 
altitudes  that  were  hazardous  to  human  pilots  (U.S.  Geological  Survey,  2011).  SUAS  can 
extend  our  capacity  to  perform  missions  in  dangerous  environments  with  limited 
personnel  due  to  their  autonomous  operating  capability  (Aeryon  Labs  Inc.,  2013). 

Sensors  such  as  MSI  and  Hyperspectral  Imaging  (HSI)  can  expand  the  capabilities  of 
SUAS.  Currently,  skin  detection  is  achieved  with  large  and  resource  intensive  sensor 
assets.  HSI  can  remotely  identify  materials,  including  skin  but  it  has  proven  difficult  to 
utilize  the  data  quickly  and  efficiently.  Situational  awareness  of  the  mission  area  may  be 
improved  by  identification  of  material  and  dismounts.  SUAS  are  quickly  deployable, 
versatile  sensor  platforms,  which  can  provide  invaluable  information  in  many  domains. 
Designing  a  sensor  system  to  detect  skin  onboard  a  SUAS  is  the  main  focus  of  this 
research. 


1.2  Background 

Data  from  aerial  platforms  demonstrated  significant  utility  in  countless 
reconnaissance,  surveying,  and  search  and  rescue  operations.  Recently,  unmanned  aerial 
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systems  (UAS)  have  proven  their  worth  in  these  domains  by  providing  precise  and  timely 
information.  Currently,  sensor  data  are  compiled  on  the  Ground  Control  Station  (GCS) 
or  the  headquarters  from  imagery  provided  over  satellite  or  radio  communications.  When 
sensor  data  are  processed,  they  usually  involve  humans  making  decisions  based  on 
imagery.  For  SUAS,  the  use  of  multiple  sensors  may  enhance  the  utility  of  the  images, 
but  may  result  in  integration  complexity  and/or  high  operator  workload  as  the  operator 
views  data  for  each  sensor.  MSI  collects  many  signatures  throughout  the  electromagnetic 
spectrum;  it  provides  the  opportunity  to  conduct  efficient  Intelligence  Surveillance  and 
Reconnaissance  (ISR),  or  SAR  (Beisley,  2012).  However,  the  additional  sensor 
dimensionality  increases  the  need  for  filters,  and  processing  time  from  computers  and 
humans.  There  are  limitations  with  the  current  approach  for  sensor  fusion  due  to 
available  communications  bandwidth,  computational  power,  and  human  responsiveness. 
By  using  more  sensors  than  necessary  to  interpret  a  target,  excessive  bandwidth  is  taken. 
Furthermore,  computational  power  for  processing  large  data  sets  may  not  be  possible  on 
small  GCS  units.  Finally,  the  ability  of  a  human  to  make  decisions  is  impaired  by 
cluttered  imagery  (Donahue,  1991). 

The  metrics  used  to  measure  the  performance  of  the  skin  detection  systems  are 
explored,  and  are  used  to  compare  a  ground  based  HSI,  with  the  proposed  aerial  skin 
detection  system.  Time  and  cost  are  key  resources  in  systems  applications;  the 
overarching  goal  of  the  SUAS  skin  detection  system  is  to  obtain  accurate  dismount 
detection,  and  minimize  system  cost.  Incorporating  a  systems  approach  to  sensor  fusion 
may  lower  costs  for  operations,  decrease  the  time  needed  to  process  data  on  the  ground, 
and  decrease  the  time  of  decision  making  for  operators  in  the  field  (Muller  &  Narayanan, 
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2009).  This  research  aims  to  measure  system  performance  and  effectiveness  of  a  SUAS 
skin  detection  system  for  dismount  detection. 

1.3  Problem  Statement 

Natural  and  manmade  disasters  jeopardize  the  safety  of  local  inhabitants,  and 
sometimes  cause  people  to  go  missing  or  astray;  first  responders  are  tasked  with  finding 
those  in  danger.  SUAS  are  mobile  platforms  that  have  proven  effective  in  many 
domains,  and  can  combine  multiple  sensors  in  a  small,  cost  effective,  and  readily 
deployable  platform.  Unique  spectral  signatures  exist  for  all  materials  and  they  provide 
another  method  to  distinguish  between  objects.  HSI  and  MSI  sensors  add  a  new 
dimension  to  images  by  increasing  the  number  of  wavelengths  collected  in  a  scene. 
However,  current  MSI  and  HSI  platforms  have  been  too  large  and  costly  to  deploy.  If 
detection  of  a  dismount  or  their  skin  can  be  accomplished  with  a  cost  effective  system 
employing  MSI  sensors  on  a  SUAS,  this  system  may  allow  an  efficient  means  for 
locating  rescue  personnel,  survivors,  or  threats.  However,  an  SUAS  presents  many 
challenges  to  imagery  that  can  make  dismount  detection  difficult.  A  multispectral  system 
on  an  SUAS  platform  requires  a  systematic  design  to  consider  any  factors  that  may 
impact  dismount  detection. 

1.4  Research  Objectives  and  Questions 

The  utility  of  the  MSI  system  for  dismount  detection  onboard  a  SUAS  is  the  main 
focus  of  this  research.  To  measure  the  utility  of  a  system,  subjective  and  objective  metrics 
will  be  used.  By  using  standardized  metrics,  the  detection  system  can  be  compared 
against  a  wide  range  of  detection  systems.  Camera  vibration,  registration  accuracy,  and 
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detection  accuracy  all  objectively  measure  system  performance.  In  tenns  of  systems 
engineering,  metrics  serve  as  Measures  of  Performance  (MOP).  Nevertheless,  the  use  of 
these  metrics  cannot  encapsulate  every  aspect  of  the  system  effectiveness.  A  Measure  of 
Effectiveness  (MOE)  evaluates  the  system  for  the  overall  mission  goal. 

Metrics  will  be  determined  that  accurately  reflect  the  utility  of  imagery  data. 
Image  Quality  (IQ)  is  largely  subjective,  but  aspects  of  it  can  be  measured  objectively 
with  metrics  identified  in  the  literature  (Gundlach,  2012).  Subjective  scales  such  as  the 
NIIRS  and  Johnson  criteria  can  be  measured  with  mathematical  relationships  to  compare 
different  multispectral  sensor  arrangements.  A  research  objective  is  to  detennine  if  these 
metrics  sufficiently  characterize  dismount  detection.  Finally,  the  overall  research  goal  is 
to  determine  the  feasibility  of  skin  detection  on  an  SUAS  platform,  this  objective  requires 
assessment  of  all  camera,  airframe,  and  communications  subsystems. 

The  investigative  questions  of  this  research  are  listed: 

1 .  Is  MSI  effective  for  dismount  detection  on  an  SUAS? 

2.  What  type  of  MSI  system  is  needed  for  an  aerial  platform? 

3.  Which  camera  parameters  are  most  critical  for  aerial  dismount  detection? 

4.  What  metrics  are  best  to  compare  dismount  detection  ability? 

1.5  Research  Focus 

This  thesis  demonstrates  the  ability  to  detect  skin  on  an  SUAS  through  the  design, 
construction,  and  evaluation  of  a  MSI  skin  detection  platform.  The  research  utilized 
known  detection  algorithms,  Commercial  of  the  Shelf  (COTS)  and  Government  off  the 
Shelf  (GOTS)  equipment,  and  aerial  flight  testing  of  the  algorithms  and  equipment.  To 
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measure  image  detection  effectiveness  of  the  proposed  system,  objective  measurements 
of  camera  pixel  values  for  IQ  and  fusion  are  compared  against  the  more  costly  SpecTIR 
HSI  and  Monocular  skin  detection  systems.  A  Receiver  Operating  curve  (ROC)  was 
utilized  as  one  metric  to  characterize  the  quality  of  the  detection  system,  and  it  can  judge 
the  effectiveness  of  the  system  in  detecting  skin  for  automated  dismount  detection 
(Beisley,  2012).  Other  objective  image  processing  metrics  were  used  for  system 
comparison,  including  the  General  Image  Quality  Equation  (GIQE),  and  the  Cumulative 
Probability  of  Blur  Detection  (CPBD)  (Narvekar  &  Karam,  2011;  Thurman  &  Fienup, 
2010).  These  objective  metrics  can  prove  useful  for  measuring  the  sensor  platfonn  utility 
when  compared  with  subjective  quality  metrics.  Each  of  these  metrics  measures  different 
qualities  and  reacts  to  unique  imagery  fusion  methods.  For  example,  one  metric  may  be 
affected  most  by  image  focus,  while  another  may  measure  edge  continuity.  In  this 
research,  the  goal  is  to  determine  imagery  platfonn  utility  with  quantifiable  metrics  and 
to  propose  a  system  that  meets  system  requirements. 

1.6  Assumptions/Limitations 

•  The  system  for  skin  detection  was  limited  to  the  sensors  available  at  AFIT, 
including  the  MSI  imager,  VIS  imager,  and  bandpass  filters. 

•  The  SUAS  used  for  the  skin  detection  was  limited  to  the  airframes 
available  at  the  AFIT  Advanced  Navigational  Technology  (ANT)  Center. 

•  For  the  time  available  for  flight  test,  a  specific  loiter  pattern  was  utilized, 
which  was  optimized  to  the  camera  parameters  (i.e.  resolution,  focal 
length) 

•  Due  to  the  cost  and  size  of  a  digital  video  acquisition  and  transmission 
system,  onboard  storage  of  imagery  was  performed  in  flight  test. 
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•  Weather  conditions  (sunlight,  rain,  wind  . . .)  were  unalterable  and  affected 
flight  duration,  autopilot  waypoint  tracking,  and  sensor  parameters. 

•  The  research  assumed  that  each  dismount  had  skin  which  was  visible  from 
the  air. 

•  A  flat  earth  model  was  used  for  processing  flight  data. 

Sensors  available  for  ISR  and  SAR  constantly  evolve  and  require  reevaluation  of 
system  parameters  such  as  slant  range,  airspeed,  and  focal  arrangement.  Some  scenarios 
may  require  sensor  systems  to  operate  at  extreme  altitudes,  temperatures,  or  winds. 

Sensor  system  requirements  are  coupled  with  airworthiness  requirements.  These 
requirements  may  govern  placement  of  sensors  and  equipment  onboard  the  airframe. 
Airworthiness  is  characterized  by  many  factors,  including  proper  center  of  gravity  (CG) 
placement,  acceptable  modifications  to  the  airframe,  adequate  power  availability,  and 
suitable  mission  flight  time.  Size  and  weight  of  the  sensor  payload,  batteries,  powertrain, 
autopilot,  servos,  and  electronics  impact  equipment  placement  and  CG.  Weight  placed 
outside  the  CG  generates  a  moment  and  can  cause  stability  issues  for  the  airframe  (FAA, 
2007).  Due  to  payload  capacity  requirements,  and  vibration  requirements,  an  electric 
version  of  the  Sig  Rascal  110  was  selected  for  the  skin  detection  system  design. 

Due  to  cost,  schedule,  and  availability,  certain  sensors  were  utilized  despite 
unfavorable  specifications  such  as  size  and  weight.  For  example,  a  filter  wheel  was 
chosen  that  added  considerable  weight  to  the  sensor  package,  so  a  lightweight  housing 
was  constructed  to  lighten  the  sensor  payload.  Range  availability  limited  the  choice  of 
weather  conditions,  and  this  was  a  corollary  of  schedule  conflict.  The  airframe, 
microcontrollers,  telemetry,  and  communications  equipment  were  configured  to  meet  the 
research  objectives  and  so  extraneous  variables  were  eliminated  where  appropriate.  For 
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example,  a  frame-grabber  and  processor  to  perform  band  math  onboard  was  outside  the 
price  range  and  a  flight  ready  configuration  of  this  equipment  would  prohibit  flight  test. 
The  wireless  data  links  available  for  video  lacked  the  bandwidth  to  transmit  data  at  an 
acceptable  rate,  and  they  were  used  primarily  for  location  estimation.  On-board  data 
loggers  were  used  to  capture  flight  video  and  telemetry  to  minimize  transmission  errors. 
A  flat  earth  Six  Degree  of  Freedom  (6DOF)  model  was  used  to  process  flight  telemetry, 
and  this  is  accurate  for  the  range  capabilities  of  SUAS.  Despite  limitations,  the  data  were 
suitable  for  a  posteriori  modeling  of  the  mission  environment. 

1.7  Materials  and  Equipment 

Multiple  components  were  utilized  for  the  SUAS,  including  an  airframe, 
communications,  video,  and  GCS  subsystems.  Each  of  these  components  was  selected  to 
meet  the  requirements  specified  in  the  literature,  and  experimentation.  The  major 
airframe  components  were  COTS  hardware  that  included  an  electric  modified  Sig  Rascal 
1 10,  a  programmable  autopilot,  and  sufficient  batteries  for  test  flights.  The 
communications  components  included  900  MHz  wireless  telemetry  modems,  and  5.8 
GHz  wireless  video  transmitters.  The  video  subsystem  components  included  a  filter 
wheel,  bandpass  filters,  a  programmable  stepper  motor  driver  and  microcontroller,  along 
with  separate  batteries  to  power  the  sensor  components.  The  GCS  included  video  and 
telemetry  receivers,  mission  planning  software,  MATLAB  and  ENVI  for  image 
processing  software,  and  frame  grabbing  hardware.  The  entire  system  was  flight  tested  at 
Camp  Atterbury,  IN  with  U.S.  Army,  U.S.  Air  Force,  and  Co-Operative  Engineering 
Services  Inc  (CESI). 
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1.8  Preview 


Chapter  II  explores  the  background  literature  for  skin  detection,  SUAS,  methods 
for  vision  processing,  and  metrics  for  imagery  analysis.  Chapter  III  presents  a 
methodology  for  data  collection  and  refers  to  the  fusion  techniques  described  in  the 
literature  review.  Chapter  IV  presents  the  results  and  discussion  of  metrics  that  were 
applied  to  experimental  data;  a  comparison  of  objective  and  subjective  metrics  is 
presented.  Chapter  V  presents  a  summary  of  findings,  critical  analysis  of  metrics,  and 
suggestion  for  future  research. 


8 


2.  Literature  Review 


2.1  Chapter  Overview 

This  literature  review  explores  background  information  needed  for  target 
detection  with  multiple  sensors.  First,  a  background  of  UAS  operation  is  presented,  to 
explain  the  nuances  associated  with  aerial  imagery.  Second,  spectral  and  spatial  image 
processing  methods  and  techniques  are  discussed.  Third,  metrics  that  can  be  applied  to 
judge  the  quality  of  dismount  detection  data  are  explained. 

2.2  Data  Fusion 

An  apparatus  that  can  autonomously  survey,  make  judgment  of  elements  in  a 
scene,  and  command  the  best  course  of  action  can  have  tremendous  impacts;  data  fusion 
can  be  effective  for  these  tasks.  Steinberg  and  Bowman  (2010),  explore  data  fusion  and  a 
model  used  by  the  DoD  to  depict  fusion.  The  Joint  Directorate  of  Laboratories  (JDL) 
established  a  model  that  is  organized  into  five  levels  of  fusion.  A  SAR  SUAS  may  serve 
as  the  basis  for  all  levels  of  fusion.  Level  0  is  the  lowest  level  of  fusion,  and  it  includes 
preconditioning  of  data.  On  the  SAR  system,  Level  0  fusion  may  involve  radiometric 
calibration  to  convert  pixel  values  to  reflectance  values.  Level  1  is  object  refinement;  it 
may  involve  fusing  feature  detectors  to  classify  objects  for  Automatic  Target  Recognition 
(ATR).  Fusing  aircraft  attitude  measurements,  GPS  locations,  and  visual  ground  features 
to  geolocate  an  image  on  an  SUAS  is  also  considered  Level  1  fusion.  Level  2  fusion  is 
Situation  Refinement.  This  fusion  may  involve  relating  objects  or  events  to  situations;  for 
example,  knowing  where  a  group  is  located,  the  status  of  their  health,  and  their  level  of 
movement  may  indicate  their  situation.  Level  3  is  Impact  refinement,  which  matches  all 
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courses  of  action  with  their  possible  impact  or  cost.  Level  3  would  combine  knowledge 
of  dismount  location  and  weather  readings  with  the  impact  of  launching  a  rescue  mission. 
Finally,  Level  4  is  process  performance  refinement  and  it  measures  efficiency  of  the 
operational  configuration  and  estimation  of  MOPs  and  MOEs;  this  level  is  often 
conducted  in  the  system  design  and  evaluation  stage  to  determine  a  system’s  goals.  In  the 
SUAS  system,  system  designers  may  realize  the  potential  issues  of  size,  time,  and  cost  by 
modeling  and  measuring  a  system  in  an  operational  setting.  For  an  SUAS  Detection 
system  to  be  useful,  it  requires  all  elements  of  data  fusion.  Since  this  research  involves 
finding  the  overall  effectiveness  of  an  SUAS  for  dismount  detection,  obtaining  the  fourth 
level  of  data  fusion  was  the  goal.  The  JDL  model  is  depicted  in  Figure  1.  (Steinberg  and 
Bowman,  2010) 
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Figure  1:  The  Joint  Directorate  of  Laboratories  (JDL)  model  for  data  fusion 
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2.3  Human  System  Integration 

Muller  and  Narayanan  (2009)  conducted  research  of  human  subjects’  responses  to 
fused  images.  This  research  focused  on  subjective  data  from  operators  on  fused  imagery 
and  collected  quantitative  data  for  time  to  identify  a  target,  accuracy  of  target 
identification,  and  confidence  in  target  identification.  The  study’s  limitation  was  the  use 
of  still  imagery,  and  inability  to  adjust  target  and  background  display  thresholds. 
Nevertheless,  the  data  suggested  that  further  research  was  needed  due  to  the  disparities  in 
subjective  detection  confidence  and  objective  decision  making  accuracy. 

Object  identification  with  multiple  sensors  is  leading  to  important  research. 
Subjective  metrics  and  objective  metrics  are  used  to  fuse  information  from  multiple 
sources,  and  these  can  impact  the  mission  integrity  (Liggins,  Hall,  &  Llinas,  2008).  There 
has  been  an  effort  to  detect  and  identify  objects  with  high  accuracy  due  to  increasing  use 
of  adaptive  autonomy  and  unmanned  aerial  systems  (UAS)  (Gundlach,  2012).  The  ability 
to  fuse  sensor  data,  and  accurately  classify  objects  in  real  time  is  crucial  for  the  operation 
of  ISR.  Integration  of  ATR  algorithms  with  human  decision  making  has  been  researched. 
This  impact  on  judgment  is  directly  related  to  the  subjective  perfonnance  measure 
proposed  by  Xydeas  &  Petrovic  (Xydeas  &  Petrovic,  2000).  Subjective  quality  is  used 
by  highly  trained  imagery  analysts  on  some  of  the  world’s  best  remote  sensing 
equipment,  despite  automatic  feature  processing.  Yoshida  (2006)  defines  subjective  IQ 
as  the  level  of  information  transmission  and  satisfaction  with  an  image  for  a  specific 
purpose.  For  an  aerial  detection  system,  mission  and  system  effectiveness  can  be 
measured  by  a  combination  of  quality  metrics. 
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Image  Quality  (IQ)  can  be  measured  in  many  ways.  Measures  include  sharpness, 
false  color,  noise,  shading,  tone  curve,  dynamic  range,  and  artifacts.  Measures  for  image 
sensor  quality  include  resolution,  spectral  response,  high/low  light  characteristics, 
defects,  angular  response,  smear,  and  lag.  Sensor  quality  is  generally  more  objective  than 
the  subjective  domain  of  IQ.  Also,  due  to  time  and  motion,  camera  shake,  motion  blur, 
jitter,  and  temporal  noise  must  be  considered.  IQ  must  be  viewed  with  a  systems 
perspective;  all  equipment  must  be  allocated  for  a  defined  purpose  as  it  is  exceedingly 
expensive  to  construct  equipment  which  provides  high  IQ  under  all  conditions. 

2.4  SUAS  and  Aerial  Imagery 

Small  UAS  have  begun  to  utilize  inexpensive  and  widely  available  sensors  to 
conduct  ISR,  and  their  utility  has  been  proven  (Beard  &  McLain,  2012;  Gundlach,  2012). 
Classification  of  objects  in  a  scenario  can  improve  the  situational  awareness  for  multiple 
stakeholders.  One  operational  scenario  was  explored  by  (Merino,  Fernando,  Martinez-de 
Dios,  Ferruz,  &  Ollero,  2006)  who  focused  on  geolocation  of  forest  fires  from  an 
unmanned  helicopter.  This  research  identified  many  factors  to  consider  with  platform 
(airship  vs.  helicopter)  onboard  processing  time,  off  board  processing,  communications, 
and  software  required  for  fire  detection.  However,  the  research  did  not  consider  the 
specific  electromagnetic  wavelengths  to  use  for  feature  extraction  or  application  to 
dismount  detections.  Nevertheless,  this  research  demonstrated  the  capability  of  SUAS  to 
perform  object  identification  in  hazardous  environments. 
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2.4.1  Image  Geometry  and  Projection 

Imagery  acquired  from  an  aerial  platform  may  require  consideration  of  aircraft 
attitude  and  position.  The  impact  of  attitude  and  position  of  an  SUAS  is  found  from 
many  sensors  onboard  the  aircraft.  The  Inertial  Measurement  Unit  (IMU)  measurements 
are  converted  to  attitude  positions  with  a  control  system  and  aircraft  equations  of  motion 
in  the  autopilot  (Beard  &  McLain,  2012).  The  autopilot  then  determines  approximate 
state  and  makes  corrections  to  its  flight  path.  A  body  fixed  camera  maintains  a  relatively 
simple  geometric  relationship  to  the  body  frame  and  inertial  frame,  compared  with  a 
gimbaled  camera.  Each  coordinate  frame  is  related  by  a  rotation  matrix,  and  they  can  be 
defined  intuitively  using  the  Euler  angle  system.  Euler  angles  are  useful  to  define  an 
SUAS  attitude,  and  they  are  combined  with  the  latitude,  longitude,  and  altitude  with 
respect  to  time  for  a  Six  Degree  of  Freedom  (6DOF)  system  (Beard  &  McLain,  2012). 
Figure  2  shows  the  directions  of  motion,  and  moments,  as  measured  by  an  IMU. 
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Figure  2:  Aircraft  body  frame  axis  direction  and  moments  that  are  measured  by  an  onboard  IMU,  which 

was  adapted  from  Bertin  (2002) 


Figure  3:  The  aircraft  Euler  angle  coordinate  system  adapted  from  Premerlani  (2009).  Euler  angle 
coordinate  systems  are  expressed  for  their  more  intuitive  nature  compared  to  Quaternions. 


Previous  research  has  used  angular  relationships  to  transfonn  imagery.  Welborn 
(2011)  developed  an  algorithm  for  determining  the  location  of  a  sensor  with  respect  to  the 


14 


inertial,  or  earth  frame.  His  model  generated  sensor  Field  of  View  (FOV)  projections  to 
the  ground  in  a  flat  earth  coordinate  system,  to  determine  the  length  of  time  that  a 
stationary  object  stays  in  the  camera  Field  of  View  (FOV).  Figure  3  shows  the  airframe 
angular  convention.  Since  telemetry  is  output  from  the  autopilot  in  the  body  frame  of  the 
SUAS,  the  algorithm  used  the  transformation  matrix  from  the  inertial  frame  to  body 
frame,  and  then  from  body  to  camera  frame.  Since  the  IMU  data  is  given  in  the  body 
frame,  one  can  then  relate  to  any  frame  of  reference  with  a  transformation  matrix.  The 
coordinate  transformation  from  the  body  frame  to  the  inertial  frame,  is  given  by  (Eq.  1). 
The  transfonnation  from  the  body  to  camera  frame  is  given  by  (Eq.  2),  assuming  that  the 
camera  is  near  the  CG  (Beard  &  McLain,  2012).  The  final  aimpoint  vector  is  given  by 
(Eq.  3).  Thus,  any  sensor  aimpoint  can  be  determined  approximately  from  the  autopilot 
telemetry.  It  is  approximate,  because  there  are  inherent  errors  and  drift  associated  with 
autopilot  sensors;  this  makes  the  data  noisy  (Beard  &  McLain,  2012).  The  sensor 
footprint  from  an  SUAS  will  change  over  time  because  the  aircraft  is  moving  at  each 
point  in  time.  Figure  4  shows  the  camera  frame  of  reference  with  respect  to  the  inertial 
reference  frame  and  the  sensor  aimpoint. 
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(Eq.  3) 


Figure  4:  The  coordinate  system  for  a  camera  equipped  airframe 

Factors  such  as  wind  and  aircraft  vibrations  may  affect  the  autopilot  pattern,  and 
decrease  total  time  on  target.  Users  leverage  the  benefits  of  SUAS  as  imagery  platforms, 
but  they  must  mitigate  the  effects  of  wind  on  the  small  airframes.  Civera  and  colleagues 
(2012)  demonstrated  the  utility  of  an  IMU  and  GPS  for  image  mosaicking;  however,  they 
also  used  computer  vision  for  feature  extraction  to  increase  positioning  accuracy.  Cho 
and  Snavely  (2013)  created  a  ground  map  from  imagery  at  multiple  positions  with 
geometric  transformations  based  on  Global  Positioning  Satellites  (GPS)  and  computer 
vision  algorithms.  The  applications  of  aerial  imagery  with  SUAS  are  limitless. 


2.5  Image  Registration 

Mapping  experts  have  long  used  registration  and  rectification  to  create  digital 
worlds.  Registration  is  the  process  of  aligning  or  overlaying  two  images  of  different 
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attitude  (Rao  &  Arora,  2004).  Rectification  is  the  geolocation  of  the  image  to  a  reference 
coordinate  system  (Liggins  et  ah,  2008).  Combination  of  images  from  multiple  sensors  is 
classified  as  Level  1  of  data  fusion  as  described  by  the  JDL  model  (Kessler,  O.,  White, 

F.,  2009;  Steinberg  &  Bowman,  2010).  When  combining  images  from  multiple  sensors, 
it  is  important  to  register  images  for  maximum  data  accuracy;  for  example,  when 
overlaying  a  topographical  map  from  HSI,  with  images  from  a  Visible  Spectrum  (VIS) 
camera,  the  pixels  from  each  sensor’s  image  must  map  by  spatial  reference  throughout  all 
channels  before  using  most  multichannel  classification  methods.  One  technique  used  to 
determine  similar  features  from  two  images  is  Scale  Invariant  Feature  transform  (SIFT). 
SIFT  is  designed  to  be  unaffected  by  scaling,  orientation,  and  mostly  unaffected  by 
lighting  and  affine  distortion  (Lowe,  2004).  This  makes  it  ideal  for  detecting  features  in 
moving  imagers,  and  the  features  can  be  applied  towards  image  registration. 

2.6  Distortion  and  Camera  Calibration 

Image  distortion  is  an  important  topic,  because  all  images  have  varying  levels  of 
distortion  or  noise  that  impairs  IQ.  Recognizing  different  image  distortion  types  is 
particularly  important  for  this  thesis,  because  the  image  fusion  methods  are  subject  to 
distortion.  Sambora  (Sambora,  2008),  described  metrics  that  attempt  to  measure 
distortion  for  multiple  aerial  sensors  including  Light  Detection  and  Ranging  (LIDAR). 

He  stated  that  imagery  distortion  can  cause  outliers  in  feature  detection  and  matching 
algorithms,  which  can  hinder  image  registration.  Feature  detection  is  leveraged  for  many 
image  registration  techniques,  and  may  be  impacted  by  noise.  Image  distortion  is  not 
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limited  to  sensor  noise,  and  can  come  from  atmospheric  disturbance,  motion  blurring,  or 
camera  optics. 

Aberrations  from  optics  may  also  account  for  distorsion.  Longitudinal  Chromatic 
Abberration  (LoCA),  is  an  occurrence  where  different  wavelengths  of  light  do  not 
converge  at  the  same  image  point  after  passing  through  a  lens  (Lluis-Gomez  & 
Edirisnghe,  2012).  This  was  discovered  when  the  lens  was  set  to  the  optimal  focus  for 
one  wavelength,  and  then  another  wavelength  was  viewed.  The  image  distortion  which 
occurred  after  the  filter  change  suggested  optical  properties  not  accounted  for  in  the  optic 
system.  The  refraction  of  light  from  the  glass  in  the  fore  optic  may  vary  according  to 
wavelength.  Also,  using  a  different  glass  type  in  the  lens  may  reduce  distortion. 
Decreasing  lens  aperture  may  lessen  LoCA,  but  a  higher  f-number  may  decrease  light 
throughput  (Lluis-Gomez  &  Edirisnghe,  2012).  Decreased  aperture  may  also  limit 
spherical  aberrations,  which  can  account  for  Airy  disks  and  additional  image  distortion 
(Mansurov,  2011). 

2.6.1  Single  Camera  Calibration 

Calibration  detennines  a  camera’s  intrinsic  and  extrinsic  parameters.  The  book 
by  Hartley  and  Zisserman  (2003)  explains  the  importance  of  a  calibrated  camera,  and  also 
gives  a  detailed  view  of  fonnulation  of  projective  geometry.  They  state  that  each  camera 
is  related  to  the  3-D  world  by  a  camera  model.  Figure  5  shows  the  pinhole  camera  model 
that  relates  an  image  to  Euclidian  space. 
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Figure  5:  Pinhole  camera  model  for  geolocation,  adapted  from  Beard  &  McLain  (2012) 
Where: 

A  =  sensor  aimpoint 
P  =  image  plane 

Pe  =  point  on  image  plane  that  is  a  detection. 
ex,y  =  distance  on  image  plane  of  detection  point 
F  =  the  distance  from  the  sensor  plane  to  the  image  plane  point  Pe. 
dXjy  =  distance  in  space  from  sensor  aimpoint  to  detection  point. 

DL  =  3-D  location  of  point  in  space 


The  importance  of  a  calibrated  camera  is  underscored  in  aerial  imagery  because 
mapping  a  pixel  location  on  the  aircraft  sensor  to  a  location  on  the  ground  requires  an 
image  plane  that  is  geometrically  similar  to  the  3-D  world. 


Finding  the  intrinsic  parameters  for  one  camera  is  accomplished  by  utilizing  point 
matching  between  multiple  images  taken  by  the  camera.  The  images  typically  consist  of 
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checkerboard  patterns,  and  points  between  checker  squares  are  extracted  by  using  corner 
detection.  Equation  4  is  the  camera  matrix  K,  and  this  characterizes  the  results  of  the 
camera  calibration  assuming  that  all  points  lie  on  a  plane.  The  camera  matrix  is  used  to 
characterize  distortion  from  sensor  so  that  images  can  be  related  to  the  actual  objects. 


K  = 


fx  ac'fx 

0  fy 

0  0 


px 

Py 

1 


(Eq.  4) 


Where: 

px,p  =  Principle  point,  or  center  point  of  image  plane 
f x  =  Horizontal  focal  length 

f  =  Vertical  focal  length 

ac  =  Skew  coefficient,  or  scaling  between  x  and  y  pixel  axis  (  >  0  for 
nonsquare  pixels) 


2.6.2  Stereo  Camera  Calibration 

Stereo  cameras  require  further  calibration  techniques  to  characterize  the 
relationship  between  image  planes.  Two  cameras  are  related  by  a  fundamental  matrix,  a 
3x3  matrix  of  rank  2,  that  can  relate  any  point,  xR ,  on  one  camera  image  to  a  point  on 
another  camera  image,  xL  (Hartley  &  Zisserman,  2003).  The  fundamental  matrix  can  be 

approximated  between  two  non-calibrated  cameras  (Bouguet,  2013).  Knowing  the 
fundamental  matrix  allows  one  to  determine  the  location  of  a  point  in  one  camera  to  the 
point  in  another.  This  case  is  simplified  for  two  parallel  cameras,  as  shown  in  Figure  6. 
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Figure  6:  Parallel  plane  stereo  cameras 


Where: 

(x,z)  =  Coordinate  frame  for  camera 
P  =  Point  on  the  image  plane  of  both  cameras 
b  =  Distance  between  both  cameras  in  the  x  axis. 
fL,fR  =  Focal  length  of  the  left  and  right  cameras  respectively 


Stereo  triangulation  can  be  used  to  determine  the  proper  registration  offset  from 
two  cameras.  This  is  possible  because  of  the  geometric  relationship: 


z’_(x'-b) 

/  xR 


(Eq.  5) 


z' 

7 


(Eq.  6) 
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z 


f-b  _f-b 


(Eq.  7) 


(xL-xR)  d 


Where: 

x',z'  =  Image  point  coordinates  in  2-D  space 
b  =  Distance  between  two  cameras 

xL,xR  =  Points  on  left  and  right  image  planes 

/  =  Focal  length,  assuming  focal  lengths  for  the  right  and  left  are 

approximately  equal. 
d  =  Disparity 

Therefore,  assuming  that  the  focal  length  of  the  right  and  left  are  similar,  we  can 
derive  the  distance  of  the  point  from  the  camera  frames  by  using(Eq.  7).  The  value  for 
disparity  can  also  be  used  to  generate  a  3-D  point  cloud  from  stereo  cameras.  However, 
if  we  know  the  distance  to  the  point,  z,  then  we  can  also  back  calculate  d  .  This  disparity 
becomes  the  image  plane  relationship  between  the  two  cameras,  or  homography. 


Stereo  camera  calibration  finds  the  precise  homography  of  the  cameras  by  using  a 
checkerboard  pattern  held  at  different  angles  and  distances  from  the  camera.  Multiple 
images  are  taken  from  both  cameras  on  the  same  target  checkerboard.  Each  image  pair 
can  be  stereo  rectified  according  to  the  projective  transform.  The  projective  transfonn 
can  be  found  from  the  stereo  camera  calibration,  and  it  consists  of  two  vectors:  rotation 
vector  and  translation  vector.  There  is  a  relationship  between  the  position  and  angle  of 
the  two  cameras,  and  it  can  be  determined  from  a  rigid  body  rotation.  Knowing  the  angle 
vector,  r  ,  and  the  translation  vector,  f ,  where 
Rotation  vector: 
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Translation  vector 


r=[_cox  coy  coz 


(Eq.  8) 


f  =  [T  Ty  TzJ  (Eq.  9) 

The  rotation  matrix  for  the  camera  can  be  related  to  the  rotation  vector  by  the 
Rodrigues  rotation  algorithm  (Pietro  Perona,  1993).  The  Rodrigues  function  outputs  the 
rotation  matrix  R  ,  with  r  as  an  input.  For  converting  from  the  rotation  vector  to  a 
rotation  matrix,  the  parameters  defined  by  (Eq.  10)  through  (Eq.  13)  are  used.  The 
Rodrigues  rotation  algorithm  is  summarized  by  (Eq.  14). 


0  —  r  —  ^ J  cox  +  coy  + 


r 

CO-  — 
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CO  = 


0  -a,  coy 
a>_  0  -a>x 

-Co,  co 0 


R  =  e 


5)6 


R  =  I  cos(0)  +  co  sin(0)  +  ( cb^  ( 1  -  cos(0)) 


(Eq.  10) 
(Eq.  11) 

(Eq.  12) 

(Eq.  13) 

(Eq.  14) 


Where: 

0  =  Euclidian  distance  between  the  rotation  vector 

co  =  Normalized  rotation  vector 

co  =  Antisymmetric  matrix 

/  =  Identity  matrix  (3x3) 
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A  rigid  motion  transformation  for  the  right  camera  to  the  epipolar  line  of  the  left 


camera  then  gives  the  coordinates  for  the  right  image;  this  relationship  is  given  by 
(Eq.  15). 


Xr=RXl+T  (Eq.  15) 

Where: 

XR,XL  =  matrix  of  pixel  coordinates  on  right  and  left  images 

The  projective  matrix  can  be  constructed  with  the  geometry  given.  Therefore,  a 
projective  transform  can  be  created  to  relate  the  images  of  the  two  cameras  at  any  slant 
distance. 

2.7  Imaging  across  the  Electromagnetic  Spectrum 

Multiple  sensors  are  used  in  ISR  and  target  detection;  cameras  can  operate  at 
different  wavelengths  in  the  electromagnetic  spectrum.  Although  most  cameras  capture 
light  in  the  Visible  Spectrum  (VIS),  it  is  because  humans  can  more  easily  detect  features 
in  this  spectrum.  Light  at  different  frequencies,  for  example,  in  the  near  infrared  (NIR), 
short  wave  infrared  (SWIR),  or  far  infrared  (FLIR),  is  more  difficult  to  utilize.  Many 
cameras  and  sensors  are  equipped  to  capture  these  types  of  light,  but  to  detect  and 
identify  different  types  of  objects,  classification  algorithms  are  often  needed.  Figure  4 
presents  the  electromagnetic  spectrum,  and  where  IR  and  VIS  sensors  are  located  with 
respect  to  their  detection  wavelength.  One  can  locate  the  different  categories  of 
wavelengths  in  Figure  7. 
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Figure  7:  Electromagnetic  Spectrum  (Ronan,  2013) 

2. 7.1  VIS  Imagers 

VIS  sensors  have  been  used  to  conduct  ISR  and  identify  targets  for  decades. 
Digital  Imagery  has  allowed  precise  measurements  of  pixel  histograms  and  color 
distributions.  A  typical  CMOS  EO  camera  collects  data  with  three  channels  that  are  in 
the  VIS.  Targets  can  exhibit  unique  signatures  for  the  VIS  such  as  matching  trees  and 
vegetation  with  green,  and  skin  or  hair  with  red.  Nevertheless,  color  images  may  provide 
irrelevant  data  for  detecting  vegetation  under  different  weather  conditions  or  climates; 
this  limits  their  use  in  ATR  applications  (Brand  &  Mason,  2000).  These  signatures  can 
still  prove  useful  in  combination  when  combined  with  other  sensors. 


2. 7.2  Multi-Spectral/Hyper spectral  Sensors 

Multispectral  imagers  (MSI)  operate  at  a  few  broadband  wavelengths  while 
hyperspectral  imagers  (HSI)  operate  in  a  few  hundred  narrow  wavelengths  on  the 
electromagnetic  spectrum  from  0.4-2. 5pm  (Rao  &  Arora,  2004).  These  imagers  create  a 
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data  cube  from  spectral  measurements.  Many  inexpensive  focal  plane  sensors  are 
responsive  up  to  the  near  infrared  (NIR),  but  expensive  components  are  needed  to 
segment  light  at  different  wavelengths  across  the  sensor  plane. 

Hsu  and  Burke  (2003)  presented  the  utility  of  multisensor  fusion  of  HSI  and 
synthetic  aperture  radar  for  image  classification.  They  presented  a  unique  Principle 
Component  Analysis  (PCA)  method  to  separate  HSI  bands  for  multiple  target  detection, 
background  classification  to  visually  delineate  targets,  as  well  as  measure  target  detection 
and  identification  perfonnance.  HSI  can  be  utilized  to  detect  objects  with  different 
physical  materials  in  real  time  with  spectral  signatures  (Hsu  &  Burke,  2003).  However, 
they  worked  with  simulated  data  and  perfect  image  registration,  which  is  far  from  actual 
operational  conditions. 

There  are  three  types  of  HSI  scanning  methods:  point  scanning,  line  scanning  or 
push  broom,  and  focal  plane  scanning.  A  point  scanning  imager  is  used  to  capture  one 
single  pixel  of  data  for  multiple  wavelengths.  For  example,  the  ASD  FieldSpec  3  can  be 
used  to  capture  highly  accurate  reflectance  data  across  the  spectrum  (350-2500nm)  but  is 
limited  to  close  range  optical  fiber  or  contact  probe  for  data  collection  (ASD  Inc.,  2012). 
The  push  broom  HSI  imager  utilizes  a  dispersion  device  such  as  a  prism  or  optical  fibers 
to  distribute  different  wavelengths  of  a  single  line  of  pixels.  An  example  of  a  line  scan 
imager  is  the  SpecTIR  LISA  built  from  dual  SpecIM  imagers  (SpecIM,  2013).  An  image 
taken  from  a  line  scanner  is  depicted  as  an  image  cube  in  Figure  8. 

A  focal  plane  detector,  or  area  scanner,  keeps  the  sensor  behind  a  bandpass  filter 
and  objective  optic  (Grahn,  Geladi,  &  Burger,  2007).  In  this  type  of  imager,  the  sensor 
receives  light  from  a  wavelength  until  the  filter  is  changed.  An  area  scan  imager  was 
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constructed  in  research  by  Peskosky  (2010),  but  used  one  entire  SWIR  camera  for  each 
wavelength  in  a  single  fore  optic  or  monocular  system.  Other  area  scan  techniques 
perform  band  separation  with  a  patterned  filter  array  in  front  of  a  sensor;  however,  these 
types  of  imagers  are  not  widely  available  (Eichenholz  &  Dougherty,  2009).  Illustrations 
of  each  HSI  mechanism  are  shown  in  the  Figure  9  and  are  adapted  from  (Grahn,  Geladi, 
&  Burger,  2007). 

The  application  of  line  scan  imagers  to  real  time  detections  may  be  difficult. 

Many  components  are  needed  for  a  line  scanning  HSI,  and  alignment  and  calibration  of 
the  optomechanics  is  critical  (Grahn,  Geladi,  &  Burger,  2007).  A  few  HSI  sensors  are 
included  in  Table  1  (Beisley,  2012)  for  comparison.  It  has  long  been  known  that  different 
materials  reflect  different  signatures  throughout  the  electromagnetic  spectrum  (Morales, 
2012).  Knowledge  of  a  particular  material’s  spectral  reflectance  can  allow  for 
classification  of  targets  by  type.  Human  skin  reflects  a  unique  signature  as  well;  in  fact, 
each  person’s  skin  has  a  unique  spectral  signature  (Nunez,  2009). 


Figure  8:  Left,  hyperspectral  image  with  363  bands  between  350-2500nm  is  displayed  as  a  radiance  cube, 
with  spectral  layers  as  depth.  Right,  radiance  measurement  from  a  single  grass  pixel  in  the  scene. 
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Figure  9:  A  few  types  of  HSI  Imagers  and  example  configurations  are  shown.  Top,  portrays  a  point  scan 


imager.  Middle,  depicts  a  line  scan  imager.  Bottom,  portrays  an  area  scanning  imager  arrangement. 
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2.8  Radiometric  Distortion  and  Calibration 


The  goal  of  spectral  imaging  is  converting  raw  images  to  unitless  values,  because 
this  allows  a  comparison  of  multiple  materials.  Properly  correcting  for  imager 
performance  and  radiometric  distortion  is  important  for  imagers  under  different 
environmental  conditions  (Beisley,  2012).  Each  camera  has  unique  performance 
parameters  for  dark  current  and  gain,  which  need  to  be  accounted  for  (SpecTIR,  2012). 
Also,  atmospheric  conditions  change  the  Sensor  Reaching  Radiance  (SRR)  in  a 
multispectral  image.  The  equation  for  total  SRR  is  given  by  (Eq.  16).  Total  SRR  is 
depicted  in  Figure  10,  where  each  type  of  sensor  reaching  light  is  denoted  by  an  arrow. 
The  literature  explains  the  conversion  of  digital  pixel  values  to  Radiance  in  greater  detail 
for  high  altitude  remote  sensing,  and  close  range  imagery  (Beisley,  2012;  NASA,  2011; 
SpecTIR,  2012). 


i . >  .  1 I  f  J,  I  l !- 


(Eq.  16) 


Figure  10:  Sensor  reaching  radiance  (SRR)  in  the  visible  to  near  IR  spectrum  is  depicted  by  arrows 
emanating  from  the  light  source,  which  is  the  sun  in  this  case.  The  sum  of  all  light  rays  entering  the  sensor 

is  the  total  SRR.  Adapted  from  Beisley  (2012) 
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Where: 


Zj  =  Path  radiance,  or  light  scattered  directly  to  sensor  FOV 
L2  =  Reflected  light  from  object 

Z3  =  Adjacency  radiance,  reflected  light  from  neighboring  objects 
p object  =  Reflectance  of  object 

Radiance  is  not  a  comparable  spectral  measurement  for  different  sensors  and 
environments,  and  readings  are  often  converted  to  normalized  reflectance.  White  and 
Gray  calibration  panels  may  be  used  to  determine  maximum  and  minimum  radiance, 
respectively.  These  are  often  used  to  determine  the  environmental  effects  (atmospheric 
vapor  interaction,  viewing  angle,  direct  reflected  sunlight  L2 )  on  electromagnetic  energy 

reflected  in  an  image  (Morales,  2012).  Most  HSI  images  are  transferred  from  radiance  to 
reflectance  by  comparing  the  manufacturer  specified  values  for  reflectance  on  the 
calibration  boards;  the  boards  function  as  truth  measurements  (Beisley,  2012).  Radiative 
transfer  (RT)  codes  are  also  used  to  convert  radiance  to  reflectance.  This  relationship 
provides  only  an  estimate  for  reflectance  and  it  is  specific  to  high  altitude  imagers.  RT 
software  such  as  MODTRAN  can  provide  a  more  accurate  estimate  for  reflectance 
conversion,  and  can  be  used  instead  of  calibration  panels.  However,  poor  parameter 
estimation  for  sun  angles,  humidity,  CCF  offset,  and  other  parameters  may  decrease 
performance  of  RT  codes  (Beisley,  2012).  Dismount  detection  with  spectral  imagers  is 
little  more  than  measuring  light  waves  reflected  from  the  dismount  of  interest.  Therefore, 
a  detection  strategy  must  account  for  the  location  of  light,  and  the  atmospheric  conditions 
in  the  scenario.  Reflectance  can  be  modeled  as  a  function  of  radiance  given  by  (Eq.  17) 
(NASA,  2011). 
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(Eq.  17) 


n-Lx-d2 

P~  ESUNX- cos  a 


Where: 

p  =  Unit-less  reflectance 

Lx  =  Sensor  reaching  radiance  \jVatts  ■  sr  lm  2nm  l  J 
d  =  Distance  measure  from  Earth  to  Sun  in  astronomical  units 
ESUNX  =  Mean  solar  exoatmospheric  irradiances  \  Watts-nf2  ■  nm  1  J 
a  =  Sun  zenith  angle  in  degrees 


2.9  Lighting  Effects 

The  incident  angle  of  light  can  impact  the  detection  of  an  object.  Koch  (2011) 
denotes  the  objects’  material  properties  as  an  independent  variable  of  the  apparent 
reflectance.  Skin  also  has  unique  reflective  properties  that  make  detection  with  HSI 
difficult.  The  relationship  between  light  angle  and  material  property  can  be  modeled  by  a 
Bidirectional  Reflectance  Distribution  Function  (BRDF),  and  the  reader  is  referred  to  the 
literature  for  more  research  on  this  subject  (Comninos,  2011;  Koch,  2011).  Different 
angles  of  reflected  light  may  translate  to  inconsistencies  in  the  detector’s  rule  based 
threshold.  Models  for  the  BDRF  have  been  derived  with  empirical  and  physically-based 
methods;  the  models  have  been  used  to  enhance  the  realism  of  3-D  computer  graphics 
(Comninos,  2011).  Figure  1 1  depicts  the  sun  angle  and  incidence  angle  of  reflection  on  a 
dismount  with  non-orthogonal  illumination  where  a  is  the  sun  zenith,  dt  is  the  angle  of 
incidence,  9d  is  the  viewing  angle  from  the  dismount  surface  nonnal.  For  any  light  based 
sensor  in  changing  environmental  conditions,  an  accurate  model  of  illumination  is  needed 
to  make  robust  detections. 
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Figure  11:  Sun  angle  affects  the  measured  skin  reflectance  on  a  dismount  from  a  sensor 

2.10  Object  Detection  Algorithms 

There  is  a  difference  between  detection  and  classification  (Shaw  &  Burke,  2003). 
In  a  scene  that  consists  of  many  different  objects,  a  classifier  would  attempt  to  code  or 
label  each  object,  while  a  detector  is  simply  looking  for  one  particular  object.  This 
research  is  concerned  with  the  detection  of  dismounts.  Many  object  detection  methods 
have  been  used  for  HSI  imagery,  but  some  require  prior,  a  priori,  knowledge  (Beisley, 
2012).  Reflectance  values  for  each  object  material  are  unique  and  large  libraries  of 
spectral  data  exist  for  earthen  materials.  Software  such  as  ENVI  can  utilize  these  spectral 
libraries  to  perform  classification  and  detection  (Excelis  Inc,  2013). 

The  methods  that  are  exploited  for  detection  of  skin  on  an  SUAS  are  outlined  in 
the  next  sections. 

2.10.1  Normalized  Difference  Index 

A  Normalized  Difference  Index  (NDI)  for  two  wavelengths  was  utilized  by 
numerous  researchers  (Beisley,  2012;  Nebiker,  Annen,  Scherrer,  &  Oesch,  2008;  Nunez, 
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2009).  The  Normalized  Difference  Vegetation  Index  (NDVI)  has  been  used  in 
agriculture  and  forestry  to  study  the  water  absorption  of  plants  (Nebiker  et  al.,  2008; 
Nunez,  2009).  The  equation  for  the  NDI  is  given  by  (Eq.  18),  where  p  is  the  spectral 
reflectance  for  each  band,  and  are  the  wavelengths  of  a  spectral  image.  The  NDI 


can  detect  alternate  materials  by  altering  the  wavelengths  \ ,  /t2 . 


NDI  = 


p(\)-p(X2) 

p(4)  +  p(T2) 


(Eq.  18) 


2.10.1,1  NDGRI  Detection 

The  basic  principle  of  the  NDI  classification  is  comparing  ratios  of  colors  to 
known  ratios  for  objects.  For  skin,  there  is  usually  a  higher  ratio  of  red  pixels  for  higher 
levels  of  skin  melanin  (Beisley,  2012).  For  plants,  there  is  usually  a  higher  green  ratio. 
Using  this  knowledge  of  green  and  red  features,  the  Nonnalized  Difference  Green  Red 
Index  (NDGRI)  algorithm  utilizes  reflectance  measurements  from  the  red  (640nm)  and 
green  (540nm)  bands  to  classify  skin  (Beisley,  2012).  Nevertheless,  these  two  bands 
alone  proved  difficult  to  differentiate  skin  or  vegetation  types  (Rao  &  Arora,  2004).  The 
ratio  for  NDGRI  is  given  by  (Eq.  19).  The  NDGRI  is  a  plausible  algorithm  for  skin 
detection  that  can  be  performed  with  low  cost  VIS  equipment. 

NDGRI  =  M54Qnm)-p(640„m)  (Eq  19 

p(540nm )  +  p(640nm) 
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2.10.1.2  NDSI  Detection 


Multispectral  imaging  can  be  used  in  collaboration  with  VIS  to  increase  detection 
probability.  To  overcome  the  limits  of  skin  detection  in  VIS  cameras,  research  has 
investigated  the  appearance  of  human  skin  and  background  objects  in  cameras  that  collect 
electromagnetic  energy  outside  the  range  of  human  vision.  While  some  flesh-colored 
background  objects  have  the  same  red,  green,  and  blue  components  as  human  skin,  they 
reflect  light  much  differently  than  skin  in  the  NIR  and  SWIR  ranges  of  the  spectrum.  MSI 
and  HSI  imagers  observe  electromagnetic  energy  across  many  spectral  channels,  creating 
multidimensional  clusters  of  skin  and  background  pixels  that  are  more  distinct  than  in 
VIS  imagery  (Beisley,  2012). 


This  computational  combination  of  VIS  and  MSI  imagery  is  used  to  classify 
objects.  The  algorithm  was  effectively  validated  with  prior  research  from  (Beisley,  2012; 
Koch,  2011;  Nunez,  2009;  Peskosky,  2010).  (Eq.  20)  is  the  Nonnalized  Difference  Skin 
Index  (NDSI)  algorithm  that  is  applied  to  the  MSI  camera  data.  The  skin  detection 
algorithm  fuses  data  in  the  NDSI  and  NDGRI  domains  by  setting  a  threshold.  The  AFIT 
monocular  and  binocular  MSI  system  utilized  this  algorithm  to  successfully  to  detect  a 
wide  range  of  sampled  skin  types  (Nunez,  2009;  Peskosky,  2010).  These  bands  identify 
key  features  in  the  spectral  signature  of  that  provide  a  method  of  segmentation  for  HSI 
imagery. 


NDSJ  =  p(l0S0nm)-  p(15S0nm) 
p(\080nm)  +  p(\580nm) 


(Eq.  20) 
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2.10.1.3  Combined  NDSI-NDGRI  Detection 


In  the  Combined  Normalized  Difference  Skin  Index  (NDSI)  algorithm,  VIS  data 
are  used  to  lower  false  detection  of  vegetation  (Beisley,  2012).  The  reason  for  using  four 
bands  in  the  electromagnetic  spectrum  for  skin  detection  is  to  suppress  false  alarms.  If 
one  used  only  the  SWIR  bands  in  the  normalized  difference  equation,  there  is  a  chance 
that  vegetation  may  be  detected  as  well.  However,  bands  in  the  VIS  spectrum,  green  and 
red,  show  significant  differences  between  vegetation  and  skin.  The  VIS  band  nonnalized 
difference  can  be  used  to  separate  vegetation  from  skin.  The  combined  score  for  the 
NDSI  and  NDGRI  is  given  by  (Eq.  21). 


dNDSI{x)  =  NDSIX  - 


f  NDGRI x  >0^1} 
{ NDGRI x  <  0  Of 


(Eq.  21) 


2.10.2  Matched  Filter  Detection 

Another  detection  algorithm  that  has  been  used  with  HSI  is  a  Matched  Filter(MF). 
Matched  Filters  have  been  commonly  used  for  detections  for  their  robustness  and 
simplicity  (Beisley,  2013).  The  matched  filter  can  provide  rapid  means  of  detection 
compared  with  other  detection  algorithms  (Excelis,  2013).  The  matched  filter  searches 
for  a  spectral  signature  of  skin,  using  a  spectral  signature  as  a  priori  knowledge.  The 
score  for  a  MF  is  defined  in  (Eq.  22)  and  a  high  score  indicates  a  close  detection 
probability. 

T 

MF(x)  =  ^~  (Eq.  22) 

s  ||  x 
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Where: 


x  =  Pixel  vector  of  all  spectral  bands 

sT  =  Spectral  signature  vector  of  the  desired  object  such  as  skin,  transpose 
||x||  =l2  Norm  of  the  vector  x 

Materials  exhibit  unique  spectral  signatures  over  the  electromagnetic  spectrum. 
MSI  and  HSI  give  us  the  ability  to  measure  the  reflectance  of  materials  in  order  to 
classify  or  detect  more  accurately  than  with  a  VIS  or  broadband  sensor.  Figure  12  shows 
the  spectral  signature  vectors  for  a  few  common  materials  from  350-2500nm  that  were 
collected  with  a  point  scan  HSI  in  the  reflectance  domain. 


Figure  12:  Measurements  were  taken  with  an  ASD  Fieldspec  Pro3  spectroradiometer  for  common  materials 
and  their  spectral  signatures.  The  vertical  lines  denote  the  bands  specifically  used  for  the  dNDSi. 
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2.11  Metrics 


Metrics  are  used  to  make  comparison  between  two  systems.  The  following 
section  describes  the  measurements  used  for  comparison  in  this  research. 

2.11.1  Detection  Measurement 

Accurate  object  detection  and  classification  is  required  for  ISR  and  SAR. 
Predicting  the  accuracy  of  a  sensor  platform  may  dictate  which  platform  is  required. 
Furthermore,  metrics  for  measuring  the  quality  of  object  detection  and  classification  may 
be  subjective  or  objective.  Judging  a  situation  based  off  these  metrics  is  most  aligned 
with  Level  2  of  the  JDL  model  (Steinberg  and  Bowman,  2010).  The  literature  described 
in  this  section  explores  which  parameters  affect  the  quality  of  skin  detection,  and  which 
factors  need  to  be  considered  for  the  environment. 

When  imaging  spectral  signatures,  each  pixel  may  contain  a  mix  of  light  reflected 
from  multiple  sources.  Positively  identifying  a  target  is  more  difficult  in  the  subpixel 
domain  (Morales,  2012).  Proper  spatial  pixel  density  is  essential  for  a  sensor  and  target 
detection  methods.  When  targeting  spectral  signatures,  an  image  must  have  sufficient 
full  pixels  on  the  target  for  most  algorithms  to  detect  (Donahue,  1991;  Morales,  2012). 
Many  metrics  have  been  developed  to  determine  if  an  image  is  suitable  for  its  specific 
purpose.  The  metrics  used  in  this  research  are  explained  below. 

2.11.2  ROC  Analysis 

Each  binary  classification  system  has  a  Receiver  Operating  Characteristic  (ROC). 
As  the  classifier  lowers  its  detection  threshold,  it  increases  the  Probability  of  true 
positives  (Ptp)  but  also  increases  the  Probability  of  False  Alarms  (Pfa).  A  ROC  Curve  is 
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generated  from  the  Probability  of  Detection  and  the  Probability  False  Alanns  at  different 
thresholds.  Therefore,  each  point  on  the  ROC  curve  can  be  written  as  a  contingency  table. 
Metrics  can  be  measured  from  the  ROC  Curve  to  compare  detection  systems.  Area 
Under  the  Curve  (AUC)  is  one  measure  that  can  quantify  the  performance  of  a  ROC 
Curve. 

2.11.3  NIIRS  and  GIQE 

Having  the  proper  pixel  density  is  important  for  detecting  targets.  A  general 
metric  forjudging  pixel  density  requirements  is  the  NIIRS  rating.  The  NIIRS  scale 
stratifies  sensor  IQ  from  (1-9)  with  9  being  the  highest  level  of  quality;  however,  it  is 
largely  subjective  and  only  quantifies  details  down  to  5cm  (Gundlach,  2012;  Kerkes  J.  P. 
&  Hsu,  2004).  An  objective  metric  was  developed  to  predict  NIIRS  ratings  based  on 
objective  scales  and  is  called  the  General  Image  Quality  Equation  (GIQE).  The  NIIRS 
scale  transforms  pixel  size  to  Ground  Sample  Distance  (GSD)  for  an  input  and  GSD  is 
given  by  (Eq.  23).  Relative  Edge  Response  (RER)  is  the  distance  it  takes  to  make  a  clear 
edge  distinction  and  it  is  given  by  (Eq.  24).  The  final  GIQE  metric  given  by  (Eq.  25)  is 
useful  for  conventional  imaging  systems,  but  may  change  when  using  a  sensor  for 
automated  target  detection,  thus  the  equation  has  been  modified  to  work  with  aberrated 
imagery  (Thurman  &  Fienup,  2010). 


GSD  =  pR  /  / 

(Eq.  23) 

RER  =  ER(  0.5  p)  -  £7?  (-0.5  p ) 

c3  G 

— —  +  c.H 

SNR 

(Eq.  24) 

NIIRS  =  c0+ct  log10  (GSD)  +  c2  log10  (RER)  + 

(Eq.  25) 

Where: 
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p  =  Sensor  pixel  pitch 

R  =  Range  of  object 

/  =  Focal  length 

ER  =  Slope  of  Normalized  edge  response. 

m  =  3x3  Sharpening  kernel  matrix 

G  =  Post  processing  noise  gain,  ( «  1  for  raw  images) 

H  =  Edge  overshoot  of  an  image 

SNR  =  Signal  to  noise  ratio  of  unprocessed  image 


2.11.4  Cumulative  Probability  of  Blur  Detection  (CPBD) 

Blur  is  encountered  in  moving  imagery  due  to  exposure  or  shutter  speed  too  slow 
to  capture  motion.  The  blur  metric  measures  the  image  distortion  due  to  motion  blur, 
focus,  or  other  aberrations.  It  utilizes  a  digital  filter  to  detect  edges  in  an  image,  and  then 
applies  a  threshold  window  for  edges  that  appear  blurred.  Another  technique  to  calculate 
blur  may  apply  a  digital  filter  for  recreating  blur  and  then  compare  the  filtered  image  to 
the  original  image.  The  principal  behind  the  metric  is  if  an  image  is  already  blurred,  then 
the  difference  between  the  original  and  distorted  will  be  less  significant.  The  benefits  of 
the  blur  metric  are  that  it  requires  no  reference  image,  in  comparison  to  the  Mean  Square 
Error,  Structural  Similarity  Index,  or  Peak  Signal  to  Noise  metrics.  The  algorithm  for  the 
CPBD  metric  is  described  in  detail  in  the  literature  (Narvekar  &  Karam,  2011). 


Table  1  shows  a  summary  of  the  metrics  used  in  this  research. 
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Table  1:  Summary  of  Performance  Metrics 


Metric 

Pros 

Cons 

NIIRS/ 

General  Image 
Quality 

Equation 

(GIQE) 

A  widely  accepted  rating 
scale  for  imagery.  (1-10), 
where  10  is  the  best  image 
quality.  Created  for  standard 
EO  cameras,  but  expanded 
to  MSI  with  GIQE 

ci  is  based  off  qualitative 

analysis.  The  scale  doesn’t 
account  for  spectral 
dimensionality. 

Receiver 

Operating 

Curve  (ROC) 
Analysis 

Quantify  detection  accuracy 
by  Pfa,  Ptp-  AUC  =  1  is  a 
perfect  detector.  Can  be 
used  to  select  optimum 
classifier  threshold. 

Less  indicative  of  quality 
when  the  ROC  approaches 
the  chance  line  (from  [0,0]  to 
[1,1]  in  ROC  space) 

Cumulative 
Probability  of 
Blur  Detection 
(CPBD) 

Image  quality  metric  that 
does  not  require  a  reference 
image.  Objective  quantity 
that  compares  well  to 
subjective  metrics. 

Low  resolution  images  or 
images  without  defined 
edges  may  not  be  accurately 
characterized. 

2.12  Summary 

This  chapter  explored  the  concepts  and  background  of  target  recognition  in  aerial 
vehicles.  Furthermore,  it  showed  a  few  basic  methods  and  metrics  to  fuse  imagery  data 
and  extract  information  for  data  fusion.  The  next  chapter  explains  the  method, 
experiments,  and  data  used  to  substantiate  the  theories  reviewed  in  this  chapter. 
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3.  Experimental  Design  and  Construction 

3.1  Chapter  Overview 

The  methods  for  accomplishing  the  research  goals  are  systematically  structured; 
the  Systems  Engineering  methodology  is  used  for  the  approach  of  design,  testing,  and 
construction.  The  steps  of  the  approach  are  listed: 

1 .  Conceptual  Design 

2.  Preliminary  Design 

3.  Detailed  Design 

4.  Evaluation  Metrics 

First,  the  conceptual  design  originates  from  the  customers  or  stakeholders  in  the 
form  of  a  problem  statement  or  Concept  of  Operations  (CONOPS).  Next,  system  level 
requirements  are  derived  from  the  problem  statement,  and  a  functional  baseline  is 
established.  Several  requirements  are  always  limited  by  the  research  scope,  or  the 
available  equipment,  timeline,  and  budget,  so  demonstrational  requirements  for  a 
prototype  system  are  defined.  Then,  the  preliminary  design  narrows  the  design  space;  an 
allocated  baseline  and  Technical  Performance  Measures  (TPMs)  are  then  established. 
From  there,  detailed  design  and  development  leads  to  definition  and  construction  of 
subsystems  and  major  elements  such  as  drawings,  parts  lists,  and  test  plans  (Blanchard  & 
Fabrycky,  2010).  Finally,  the  metrics  used  for  validating  the  requirements  are  explained. 
The  scope  of  this  study  precludes  further  exploitation  of  the  systems  lifecycle,  and  thus 
for  this  research  the  final  level  of  “utilization”  is  not  considered. 
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3.2  Conceptual  Design 

The  goal  of  systems  engineering  is  to  execute  a  process  that  creates  an  end 
product  or  tool  that  is  useful,  efficient,  and  robust.  Using  this  methodology,  the  problem 
was  initially  defined:  measure  the  feasibility  of  a  skin  detection  multispectral  sensor  on 
an  SUASfor  Search  and  Rescue  operations.  The  conceptual  requirements  derived  from 
the  problem  scenario,  include: 

•  Operation  in  an  austere  environment  with  minimal  operators; 

•  Range  and  speed  that  allows  SAR  operations; 

•  Sufficient  slant  distance  for  extreme  environments; 

•  Adequate  pixel  density  for  detecting  skin  on  a  dismount; 

•  Imagery  that  allows  positive  identification  of  dismounts. 

3.2.1  System  Requirements 

System  level  and  technical  requirements  were  then  derived.  An  airframe  that 
could  withstand  the  environmental  conditions  of  a  forest  fire,  or  thunderstonn  would  be 
ideal  for  austere  operation.  Minimal  equipment  on  the  GCS  would  make  the  system 
easier  to  use  and  require  less  operators.  An  airframe  power  delivery  system  that  provides 
ample  range,  speed,  and  flight  time  for  coverage  of  large  areas  helps  to  achieve  the  SAR 
mission.  A  multispectral  imagery  system  presents  a  valid  method  to  detect  skin  and 
dismounts.  For  real  time  dismount  detection  on  an  aerial  platfonn,  an  area  scanner 
system  presented  the  best  method  of  spectral  imaging.  Area  scanning  imagers  minimize 
registration  processing  to  create  an  image  cube,  and  having  simultaneous  imagery  of 
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different  spectral  bands  makes  the  system  viable  for  near  real-time  detections.  Slant 
distance  requirements  are  related  to  the  nature  of  the  enviromnent,  and  sufficient  slant 
distance  is  required  to  keep  the  airframe  flight  worthy  in  high  radiation  or  heat 
conditions.  Peskosky  (2011)  proposed  a  215m  slant  range  and  5cm  pixel  size 
requirement  for  SAR.  Pixel  size  was  determined  from  pixels  needed  on  a  face  or  hand 
for  detection,  and  this  is  shown  on  Figure  13.  Finally,  imagery  that  meets  subjective  and 
objective  quality  standards  is  required  for  operators  and  ATR;  adequate  frame  rates  and 
high  true  positive  detection  rates  allow  more  information  to  be  gathered  from  a  scene 
such  as  the  size,  number,  or  location  of  dismounts. 


Figure  13:  Red  area  denotes  full  skin  pixels,  and  yellow  area  denotes  partial  skin  pixels.  Left  shows  a 
representative  dismount.  Middle,  shows  the  most  dense  pixel  distribution  on  average  sized  face  and  hand. 

Right,  shows  the  least  dense  pixel  distribution  on  face  and  hand.  A  grid  size  of  5cm  was  used. 

3.2.2  Flight  Expedient  Prototype  Requirements 

Due  to  the  scope  of  this  research,  the  flight  expedient  prototype  system  could  not 
satisfy  all  requirements  set  forth  by  the  operational  requirements.  The  first  difference 
was  the  airframe.  The  available  prototype  airframe  exhibits  comparable  performance  to 
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other  SUAS  in  its  size  category  for  speed  and  altitude;  however,  the  airframe  did  not 
meet  the  requirements  for  operation  in  an  austere  environment.  The  prototype  airframe 
was  limited  to  15-20  minutes  flight  time  and  constructed  with  materials  that  are  less  than 
favorable  for  forest  fire  SAR.  The  second  difference  was  the  MSI  sensor.  Due  to  the 
cost  and  availability  of  area  scanners,  a  high  frame  rate  and  fast  changing  filtering  setup 
replaced  a  multi-sensor  system.  For  the  single  sensor  system,  to  vary  the  bandwidth  of 
light  to  the  sensor  a  variable  Thin  Film  Transistor  (TFT)  filter  was  considered.  Also,  a 
patterned  filter  array  was  considered.  However,  due  to  costs  and  modification  needed  for 
each  of  these  filtering  systems,  a  filter  wheel  with  multiple  bandpass  filters  was  chosen 
for  this  research.  A  filter  wheel  was  within  budget,  and  materials  were  available  for 
construction.  Achieving  high  frame  rates  with  an  alternating  filter  element  had  new 
challenges  that  are  discussed  in  this  chapter.  The  third  difference  between  operational 
requirements  and  prototype  configuration  was  the  video  processing  architecture.  Real 
time  detection  ability  was  limited  due  to  the  wireless  transmission  requirements,  and  so 
data  was  stored  onboard  the  autopilot  and  digital  video  recorders.  The  fourth  difference 
was  operating  altitude  and  slant  distance.  The  slant  distance  requirement  could  not  be 
met  due  to  optics  availability;  therefore,  shorter  distances  were  used  for  testing.  Altitude 
for  test  flights  ranged  from  30-60  meters,  similar  to  the  operational  testing  done  at  Eglin, 
AFB  for  small  forest  fire  monitoring  (Aeryon  Labs  Inc.,  2013).  Pixel  size  was 
determined  to  vary  between  5.0  and  8.0  cm  with  the  available  optics.  This  was  within  the 
bounds  of  the  system  requirements.  Flight  expedient  prototype  requirements,  which 
satisfied  all  achievable  operational  goals  with  the  budgeted  resources,  are  outlined  in  . 

Table  2. 
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Table  2:  Flight  expedient  prototype  requirements  for  SUAS  MSI  system 


Requirement 

Value 

Airframe 

Endurance 

10-20  min 

Speed 

10-20  m/s 

Propulsion 

Battery-Electric 

Power/Weight 

1 10-220  watts/kg 

Weight 

<10kg 

Altitude 

30-50  m 

Payload 

<2  kg 

Launch  Mechanism 

Conventional  /  Hand  launch 

Recovery  Mechanism 

Conventional  /  Grass 

Optical 

Slant  Distance 

42-100  m 

Pixel  Size 

5. 0-8.0  cm 

FOV1 

25°  diagonal 

Optical  Design 

Refractive 

Frame  Rate 

1  fps 

Sensor  pitch 

25pm  (4/3”  sensor) 

Spectral 

Wavelength  Sensitivity 

350-1600  ntn 

Bands  CWL" 

540,  660,  1050,  1550  ±8nm 

Bandpass  FWHMJ 

10  ±  2nm 

3.3  Preliminary  Design 

3.3.1  Sensor  considerations 

The  SWIR  camera  was  chosen  based  on  availability  and  compatibility.  Peskosky 
(2010),  utilized  the  Goodrich  SWIR  cameras  to  create  a  4  sensor  skin  detection  platfonn. 
The  Goodrich  SWIR  camera  can  detect  wavelengths  from  600-1700nm;  it  includes 
multiple  user  adjustable  parameters  such  as  gain,  exposure,  bad  pixels,  and  frame  rate. 
The  camera  has  the  option  to  output  digital  and  NTSC  video  as  well.  The  sensor  angle, 

1  Field  of  View 

2  Center  Wavelength 

3  Full  Width  Half  Max 
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slant  distance,  and  altitude  was  then  chosen  based  on  airframe  considerations.  Next,  a 
lens  size  was  chosen  based  on  HSI  sensor  parameters.  This  lens  must  allow  adequate 
pixel  density  from  a  safe  aerial  slant  distance;  slant  distance  is  measured  from  the  camera 
lens,  to  the  region  of  interest.  Field  of  View  (FOV)  was  also  attained  from  lens  size. 

After  determining  the  SWIR  camera  configuration,  VIS  camera  selection  was  initiated. 

The  investment  costs  of  testing  and  components  were  considered  in  this  research. 
The  cost  of  VIS  cameras  was  not  nearly  as  prohibitive  as  the  SWIR.  Most  VIS  cameras 
surpass  the  SWIR  in  resolution,  are  smaller  in  size,  and  require  less  power.  Integration  of 
the  VIS  camera  was  not  as  challenging  as  the  SWIR.  Nevertheless,  any  multimodal 
imaging  system  benefits  from  similar  intrinsic  and  extrinsic  properties  for  image  fusion. 
The  VIS  camera  provided  onboard  data  storage,  good  IQ,  low  cost,  and  a  small  fonn 
factor. 

The  algorithm  available  for  skin  detection  was  then  determined.  From  the 
literature,  a  proven  algorithm  for  skin  detection  is  the  NDSI  and  NDGRI.  This  approach 
utilizes  4  bands  in  the  VIS  and  SWIR  light  spectrum,  495-570nm,  620-750nm,  1050- 
1  lOOnm,  and  1550-1600  nm.  For  the  first  two  bands,  a  VIS  camera  can  be  used  to  collect 
data.  For  the  second  two  bands,  a  SWIR  camera  can  be  used  to  collect  data.  Several 
methods  have  been  used  to  separate  and  collect  these  electromagnetic  spectrums,  and 
they  are  described  in  Section  2.  Restricting  light  to  1050  and  1550  wavelengths  with  the 
SWIR  required  at  least  two  bandpass  filters.  This  detection  method  thus  dictated  the 
optical  equipment  that  would  need  to  be  added  to  the  sensor  package. 

Each  optical  filter  alters  the  electromagnetic  signal  (light)  transmittance.  In  a  filter 
wheel,  each  optic  is  mounted  to  a  rotating  wheel  which  passes  in  front  of  the  sensor 
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plane.  Using  multiple  filters  to  take  images  of  the  same  object  creates  spectral  depth,  and 
allows  construction  of  an  image  cube.  However,  constructing  an  image  cube  with  a 
moving  vehicle  requires  additional  processing.  This  drove  technical  requirements  for  the 
filter  wheel.  The  filter  wheel  must  be  capable  of  switching  between  bands  fast  enough  to 
allow  similar  features  to  appear  in  each  subsequent  image  for  computer  vision 
registration.  After  experimentation  with  different  values  of  video  registration  overlap,  a 
50%  overlap  consistently  provided  enough  common  features  for  images.  Proper 
registration  accuracy  was  highly  correlated  to  detection  accuracy.  Since  sensor  payload 
required  for  skin  detection  dictated  airframe  selection,  the  sensor  integration  required 
careful  consideration  for  use  on  the  parameters  of  the  aerial  platform. 

3.3.2  Airframe  Selection  and  Modification: 

There  were  a  few  options  for  airframes  available  at  the  AFIT  Advanced 
Navigation  Technology  Center  and  Systems  Engineering  Departments.  The 
AeroVironment  Raven,  Bixler  Super  Sky  Surfer,  and  Sig  Rascal  110  were  all  considered 
as  test  beds.  All  were  in  the  same  Group  I  category,  therefore  none  exceeded  9  kg  (201b) 
gross  weight,  exceeded  51  m/s  (100  KIAS),  or  would  exceed  365m  (1200ft)  AGL 
operating  altitude.  These  categories  are  defined  in  the  AFI  1 1-502  (Department  of  Air 
Force,  2012)  .  Among  the  requirements  considered,  the  most  notable  were: 

1 .  Capability  for  holding  a  sensor  platform  of  known  weight  and  size 

2.  Environmental  robustness  of  the  aerial  system  to  withstand  varied  conditions 

3.  Capability  of  the  airframe  to  sustain  flight  for  takeoff,  navigation  to  a  known 
waypoint,  and  return  flight 
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First,  the  smallest  capable  airframe  of  holding  the  sensor  payload  was  considered; 
this  airframe  is  the  Sig  Rascal  1 10.  This  airframe  is  capable  of  carrying  a  2.5  kg  payload, 
and  was  known  as  a  stable  platform  for  performing  SUAS  research  (Jodeh,  2006). 
However,  there  are  several  variations  of  the  Sig  Rascal.  Gasoline  and  electric  variants 
have  certain  tradeoffs  for  flight  time,  weight,  operator  usability,  noise,  and  payload. 

Since  fixed  body  cameras  are  sensitive  to  vibrations,  the  best  airframe  for  a  skin  detection 
sensor  system  would  minimize  vibrations.  The  electric  powered  Rascal  variant  had  less 
vibration  than  the  gasoline  powered  variant,  while  still  allowing  adequate  flight  time  and 
payload  capacity  (Giacomo,  2012).  The  one  requirement  the  Rascal  could  not  achieve 
Considering  all  tradeoffs,  the  electric  Rascal  was  chosen  as  the  airframe  for  research. 

A  preliminary  flight  test  was  done  without  the  MSI  sensor,  and  it  was  useful  for 
characterizing  the  airframe  speed,  stability,  and  oscillatory  performance  during  flight. 
Design  choices  were  made  with  flight  data  from  a  representative  SAR  flight  pattern.  The 
airframe’s  speed  dictates  how  fast  objects  move  through  the  sensor’s  FOV.  Therefore, 
the  speed  of  the  airframe  determined  the  requirement  for  frame  processing  rate;  because 
there  was  only  a  short  amount  of  time  to  capture  images  of  the  same  object.  A  flight  test 
was  then  conducted  to  measure  the  cruise  speed,  and  adjust  the  autopilot  control  gains  for 
a  simulated  sensor  payload.  Tuning  the  gains  for  coordinated  flight  paths  in  windy 
conditions  was  crucial  for  autopilot  navigation.  Testing  data  was  then  used  for  deriving 
other  prototype  system  requirements. 


48 


Overlap%  and  Pixel  Size 


Figure  14:  Comparison  of  different  sensor  angles  using  test  flight  telemetry  at  32m  AGL.  For  a  filter 
change  rate  of  0.5  Hz,  the  corresponding  frame  overlap  %,  working  distance  in  meters,  and  pixel  size  for 

SWIR  and  VIS  sensors  in  mm  is  displayed. 


Flight  telemetry  could  be  used  in  lieu  of  further  testing  to  predict  camera 
movement  and  geometry.  A  Monte  Carlo  analysis  was  applied  to  straight  line  passes  of 
the  rectangular  or  “race  track”  flight  pattern  to  predict  sensor  FOV  across  different  sensor 
angles.  Figure  14  shows  a  few  parameters  changing  as  a  function  of  sensor  declination 
angle.  The  green  and  red  lines  show  camera  pixel  size  in  millimeters  from  the  image 
plane  on  the  ground.  It  can  be  seen  that  the  VIS  camera  pixels  are  significantly  smaller 
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than  the  SWIR,  and  this  makes  sense,  due  to  the  resolutions  of  each  camera.  The  purple 
dotted  line  shows  the  working  distance,  which  is  the  distance  from  sensor  to  ground.  The 
blue  line  shows  the  percentage  of  overlap  for  the  SWIR  FOV  given  2  frames  per  second. 
The  maximum  attainable  overlap  appeared  near  -15°  sensor  declination  angle.  However, 
the  smaller  magnitudes  of  sensor  declination  also  increased  pixel  size.  The  simulation 
was  used  to  determine  the  optimum  0sensor ,  which  was  -30°  for  the  2  fps  SWIR  image 
acquisition  rate. 

3.4  Detailed  Design 

Detailed  design  involved  assembling  the  components  to  subsystems,  and  then 
testing  each  separately.  Integration  testing  was  then  completed  with  the  subsystems 
during  ground  testing.  These  components  all  composed  the  SUAS  Multispectral 
Detection  Platfonn  as  shown  in  Figure  15.  This  allowed  key  components  and  settings  to 
be  changed  before  operational  testing. 
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Figure  15:  Physical  system  architecture  for  SUAS  MSI 
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3.4.1  MSI  Video  Subsystem 

For  the  SWIR  bands,  a  Goodrich  InGaAs  sensor  was  used.  Each  setting  in  the 
Goodrich  SWIR  camera  needed  to  be  adjusted  for  use  in  the  video  subsystem.  Settings 
that  could  alter  the  skin  detection  ability  include:  digital  gain,  look  up  table  (LUT), 
contrast  factor,  pixel  offset  value,  bad  pixel  mapping,  and  enhancements.  As  described  in 
Section  2,  radiance  conversion  requires  that  the  SWIR  camera’s  signal  processing  system 
first  captures  light  on  an  InGaAs  sensor  array.  Next,  it  applies  gain,  and  analog  offset 
video,  before  processing  by  the  analog  to  digital  convertor.  After  digital  conversion,  the 
video  is  modified  for  bad  pixel  offset/gain/replacement,  user  enhancement  controls  (i.e. 
contrast),  Lookup  table  control,  and  then  it  is  output  to  a  digital  or  analog  port. 

For  the  VIS  bands,  a  hobbyist  camera  was  used.  The  camera  settings  of  the  VIS 
were  less  sensitive  than  the  SWIR  because  there  were  no  moving  components  needed  to 
separate  wavelengths.  The  VIS  camera  segments  data  to  660nm,  540nm,  and  470nm 
channels  with  Bayer  patterning  (Omnivision,  2009).  Specifically,  the  Auto  Gain  Control, 
exposure,  pixel  map,  were  able  to  be  kept  at  factory  settings  without  modification.  For 
the  particular  camera  selected,  the  video  signal  was  output  to  NTSC  in  real-time; 
however,  the  full  sensor  resolution  was  not  available  with  this  option.  To  utilize  Full 
High  Definition  (FHD)  VIS  video,  onboard  storage  was  used  for  detections. 

3.4.1. 1  Lens  Selection 

For  the  SWIR  sensor,  a  compact  fixed  focal  length  lens  was  selected.  This  lens 
form  factor  was  chosen  for  its  small  size  and  weight  on  the  aerial  mounting  platform. 

The  lens  focal  length,  which  was  35mm,  was  chosen  for  proper  pixel  density  and  FOV. 
The  fused  silica  lens  type  was  chosen  for  its  high  transmission,  which  is  a  measure  of 
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light  throughput  in  a  material;  index  of  refraction,  which  is  a  ratio  of  the  speed  of  light  in 
a  vacuum  to  light  in  the  material;  and  dispersion,  which  is  a  measure  of  the  change  in  the 
index  of  refraction  when  wavelength  changes  (Edmund  Optics,  2013). 

The  sensor  format,  resolution,  and  spectral  sensitivity  of  the  VIS  camera  required 
unique  lens  consideration.  Research  indicated  that  a  low  pass  filter  would  block  IR 
wavelengths,  while  transmitting  the  VIS  wavelengths  (Peskosky,  2010).  IR  light  can 
produce  inaccurate  reflectance  conversions  from  VIS  imagery.  Therefore,  a  low  pass 
filter  was  installed  in  front  of  the  sensor  to  increase  color  response  of  VIS  wavelengths. 
Due  to  the  sensor  resolution,  the  lens  chosen  for  the  RGB  camera  had  a  focal  length 
adjustable  between  9-22mm.  A  focal  length  around  12mm  was  chosen,  and  provided 
double  the  pixel  size  of  the  SWIR  camera  pixels;  this  required  images  to  be  scaled  when 
registering  with  SWIR  images. 

3.4.1.2  Data  Logging  and  transmission 

The  SWIR  imager  data  logger  was  an  analog  input  DVR.  The  native  resolution  of 
the  SWIR  imager  is  640x5 12,  and  the  analog  output  was  NTSC  (640x480).  The  video 
was  stored  on  Secure  Digital  (SD)  cards,  and  recorded  at  a  maximum  resolution  of  D1 
(704x576).  Due  to  the  format  conversion  to  NTSC,  the  maximum  recorded  resolution  on 
the  SWIR  system  was  640x480,  so  scaling  corrections  were  needed  on  the  video  during 
processing.  Ideally,  a  digital  frame  grabber  or  transmitter  is  used  to  allow  high  speed 
triggering  and  frame  acquisition.  However,  the  size,  weight,  and  cost  associated  with  a 
digital  solution  restricted  use  of  digital  storage  for  the  SWIR  channel.  An  analog  video 
transmitter  sent  video  to  the  GCS  for  operators  to  determine  appropriate  flight  patterns. 
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3.4.1.3  Filter  Wheel  Design 

The  operational  requirement  of  the  MSI  system  is  the  ability  to  capture  images 
over  multiple  wavelengths.  For  this  requirement,  a  filter  wheel  with  different  filters  was 
chosen  for  cost  and  availability.  Knowing  the  proper  filter  position  may  be  important  for 
any  configuration  with  multiple  filters.  For  the  feature  detection  algorithm  chosen,  using 
the  correct  wavelength  is  essential.  The  Hall  Effect  feedback  system  can  be  used  to 
sense  filter  changes,  locate  the  position  of  the  filter  wheel,  or  monitor  wheel  position 
while  it  is  moving.  Furthermore,  the  algorithm  employed  to  detect  skin  in  this  research  is 
sensitive  to  the  difference  between  sequential  frames  that  are  captured  on  the  SWIR 
imager.  Thus,  a  derived  technical  requirement  was  adequate  filter  changing  speed.  The 
design  minimized  time  between  band  changes  to  improve  image  registration  accuracy. 

A  filter  wheel  system  is  constructed  of  multiple  parts.  First,  there  is  a  filter  wheel 
to  hold  the  bandpass  filters.  The  wheel  has  six  positions  for  filters,  and  any  type  of  filter 
can  be  inserted;  however,  bandpass  filters  were  used  and  each  one  enabled  transmission 
of  light  at  a  specific  wavelength.  Second,  there  is  the  stepper  motor,  which  rotates  the 
filter  wheel;  the  motor  must  be  able  to  switch  filters  at  an  adequate  speed,  and  draw  as 
little  current  as  possible.  Third,  there  is  the  filter  positioning  sensor  shown  in  Figure  16. 
The  sensing  system  is  constructed  of  Hall  Effect  sensors  and  magnets  that  can  tune  the 
location  and  speed  of  the  rotating  filters.  Fourth,  there  is  the  motor  driver  shown  on  on 
Figure  17,  left.  The  driver  is  built  with  a  transistor  array  which  sends  pulses  of  current  to 
a  motor’s  coils.  Increasing  the  frequency  of  pulses  causes  a  stepper  motor  to  increase 
rotation  speed.  Fifth,  is  a  microcontroller  that  controls  the  speed  and  duration  of  motor 
pulses.  The  microcontroller,  which  is  shown  on  the  right  in  Figure  17,  can  be 
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programmed  to  send  a  velocity  profile  to  the  motor  driver,  read  data  from  the  position 
sensing,  or  transmit  data  to  the  autopilot  with  serial  communication.  Sixth,  is  the  filter 
wheel  housing  shown  in  an  exploded  view  on  Figure  18.  The  filter  wheel  housing  was 
constructed  from  lightweight  polymer  using  additive  manufacturing  techniques  in  order 
to  save  weight  and  create  detailed  features  that  are  expensive  and  time  consuming  on  a 
conventional  milling  machine.  The  motor,  filter  wheel,  and  sensing  system  are  mounted 
on  the  housing,  which  provides  protection  from  the  environment. 


Figure  16:  Left,  Hall  Effect  sensor  (Melexis,  2006).  Right,  shows  earth  magnet  and  direction  of  field. 
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Figure  18:  Left,  depicts  an  rendering  of  the  filter  wheel  housing  model.  Right,  depicts  a  wireframe  model. 


There  are  several  technical  requirements  for  the  stepper  motor  and  driver;  size, 
weight,  and  filter  location  accuracy  of  the  system.  A  COTS  system  for  switching  filters 
with  a  filter  wheel  is  the  ThorLabs  FW-103H  filter  wheel,  and  a  Thor  BSC- 101  stepper 
motor  driver.  To  move  the  sensor  within  a  prescribed  amount  of  time,  the  driver  must 
supply  sufficient  voltage  to  the  stepper  motor.  The  motor  driver  can  deliver  a  trapezoidal 
velocity,  or  constant  speed  profile  to  the  stepper  motor  (Thorlabs,  2010).  The  driver  is 
exceptionally  accurate  due  to  the  Hall  Effect  sensors  which  indicate  filter  wheel  location 
and  speed,  however,  the  system  is  prohibitively  large  and  heavy  for  an  SUAS  application. 
For  this  reason,  a  small  stepper  motor  driver  was  built. 

Stepping  the  motor  to  specified  velocities  and  accelerations  required  calibration  to 
ensure  proper  filter  position.  After  each  step,  small  inconsistencies  in  velocity  and 
acceleration  can  accumulate  over  multiple  steps,  and  this  may  cause  misalignment.  To 
control  the  filter  wheel  in  a  closed  loop  system,  sensors  were  used  to  provide  a  feedback 
loop.  Hall  Effect  sensors  that  measure  magnetic  flux  density  were  installed,  and  used  to 
determine  filter  wheel  position  and  alignment.  The  sensors  communicate  with  the  driver 
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by  sending  logical  values  of  LOW  or  HIGH  if  there  is  magnetic  flux  greater  than  the 
threshold  amount.  The  US5881  Hall  Effect  sensors  that  were  used  in  construction  of  the 
fdter  assembly  have  a  threshold,  B,  of  20mT,  at  5V  operating  voltage  (Melexis,  2006). 
The  relationship  for  detennining  gap  is  given  by  (Eq.  26)  (Supermagnete,  2013). 
Parameters  from  Table  3  are  for  the  chosen  magnets  and  used  in(Eq.  26).  Using  Finite 
Element  Method  Magnetics  (FEMM),  the  required  distance  between  the  filter  wheel  and 
Hall  effect  sensor  z,  was  determined  to  be  3.5  mm.  Figure  17  visualizes  the  magnetic  flux 
distribution  for  the  Neodymium-Iron-Boron  (NeFeB)  magnet  using  visualization  model 
using  FEMM  (Meeker,  2013). 


C  = 


L  ■  W 

2zy/4z2+L2  +  W 2 


L-W 

D  = -  -  (Eq.  26) 

2(D  +  z)sj4(D  +  zf  +L2  +  W2 

B  -  — [arctan(C)-arctan(D)] 


Table  3:  Properties  and  dimensions  for  NdFeB  N42  magnet 


B 

r 

Residual  flux  density 

1.28-1.32  Tesla 

L 

Length 

2.0  mm 

W 

Width 

2.0  mm 

D 

Height 

1 .0  mm 

z 

Distance  from  pole  face 
on  symmetry  axis 

3.5  mm 
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|5.829e-001 :  >6.13te-001 
5.525e-001 : 5.829e001 
5.220e-CDl :  5.525e-001 
4.915e-001 : 5.220e-001 
4.611e-001 : 4.915e-001 
4.306e-001 : 4.611e-001 
4.001e-001 : 4.306e-001 
3.697e-001 : 4.001e-001 

_  3.392e-001 : 3.697e001 

_  3.087e-001 : 3.392e-001 

_ Z783e-001 : 3.087e-001 

_ Z478e-001 :  Z783e001 

_ Z173e-001 :  Z478e-001 

_  1.869e-001 :  Z173e-001 

_  1.564e-001 : 1.869e-001 

_  1.259e-CDl :  1.5&te-001 

_ 9.546e002 : 1.259e-001 

_  6.499e-002  :  9.546e002 

_  3.452e-002 : 6.499e-002 

□  <4.056e-0CB :  3.452e-002 
Density  Rot  |B|,  Tesla 


Figure  19:  Left,  The  magnetic  flux  distribution  for  filter  wheel  position  indicator  magnets  is  visualized. 


Distance  from  magnet  surface  (mm) 

Figure  20:  The  magnitude  of  magnetic  field  is  displayed  as  a  function  of  distance  from  magnet  surface  on 
the  magnet  vertical  (+j)  axis.  The  proper  distance  between  the  magnet  and  sensor  was  required  for  the 

US5881  to  activate. 


The  stepper  driver  supplied  by  Rugged  Circuits  was  integrated  to  the 
microcontroller,  which  is  based  off  the  Arduino  interface.  The  driver  was  programmed  to 
operate  in  constant  speed,  and  the  AccelStepper  library  allowed  for  custom  velocity 
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profiles  (McCauley,  2013).  Thorlabs  utilized  a  trapezoidal  velocity  profile  to  achieve 
consistent  steps  (2010);  however,  experiments  showed  trapezoidal  velocity  profiles  were 
less  predictable  as  battery  charge  decreased.  The  effect  of  battery  life  on  both  velocity 
profiles  is  illustrated  in  Figure  2 1 .  The  trapezoidal  stepper  pattern  was  applied  and  tested 
on  the  Arduino  stepper  driver;  however,  accuracy  of  steps  was  not  consistent.  For  this 
reason,  the  filter  wheel  was  programmed  to  operate  at  a  constant  speed.  Hall  Effect 
sensors  were  programmed  to  report  filter  wheel  position. 

The  stepper  driver  was  programmed  to  first  output  a  signal  when  the  zero  position 
was  reached,  and  then  output  a  signal  when  any  filter  was  directly  in  front  of  the  camera 
optic.  This  allowed  for  filter  position  to  be  known  for  real  time  processing  of  SWIR 
footage.  Without  the  proper  filter  location  sensing  equipment,  the  triggered  image  may 
inherit  multiple  types  of  aberrations  (Edmund  Optics,  2013). 


Velocity 


♦  *  Frame  Trigger 

-  Constant  Speed 

-  Trapezoidal  Profile 


Figure  21:  Left:  Trapezoidal  velocity  profile  compared  with  constant  speed  for  stepper  motor  actuation. 
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3.4.1.4  Mounting  Design 

The  camera  mount  was  constructed  from  lightweight  materials.  The  baseplate 
and  aircraft  mounting  assemblies  were  machined  from  aluminum  and  allowed  for  30  -  45 
degree  sensor  angle,  9sensor  adjustment.  The  camera  and  fdter  wheel  components  were 

mounted  with  vibration  damping  material  in  compression.  The  baseplate  was  affixed  to 
the  airframe  under  the  CG  and  the  APM  autopilot.  The  mount  was  first  constructed  as  3- 
D  solid  models  to  detennine  placement,  weight  balance,  and  where  additional 
reinforcement  was  required  on  the  airframe.  The  Sig  Rascal  110  3-D  model  was  the 
same  one  used  for  Hardware  in  the  Loop  (HIL)  simulations  in  FlightGear  (2013).  The  CG 
on  this  airframe  was  calculated  from  research  on  the  gas-powered  Rascal  1 10  by  (Jodeh, 
2006).  Figure  22  depicts  the  location  of  the  CG  on  the  Sig  Rascal  110,  and  the 
components  that  were  added  to  the  model  with  respect  to  the  thrust  line,  shown  as  a  blue 
line,  and  the  center  of  lift,  shown  as  the  red  line.  A  fixed  body  camera  allowed  for  easier 
modeling  of  the  sensor  aim  point,  because  the  camera  frame  is  subject  to  similar  rotations 
and  translations  as  the  autopilot,  which  outputs  telemetry  data.  The  models  were 
produced  during  the  Detailed  Design  Phase  to  accurately  predict  weight  and  size  of  the 
payload,  and  to  plan  reinforcements  to  the  Rascal  110  fuselage.  Figure  23  shows  the 
detailed  model  of  the  MSI  assembly.  Figure  24  shows  the  model  of  the  entire  airframe 
with  sensor  range  of  adjustment. 
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Figure  22:  Location  of  MSI  sensor  components  relative  to  the  CG,  which  is  located  near  the  intersection  of 
the  thrust-line  and  Center  of  Lift  of  the  Sig  Rascal  110. 
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Figure  23:  MSI  Sensor  assembly 
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Figure  24:  3-D  Model  of  camera  assembly  onboard  Sig  Rascal  110  a  ray  depicts  the  sensor  angle  degrees  of 
movement.  The  mount  is  adjustable  from  30-45  degrees  of  declination. 


Figure  25:  MSI  sensor  mounted  to  the  airframe. 
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Figure  26:  Test  Sig  Rascal  1 10  airframe  and  sensor  payload 

3.4.2  Stereo  calibration 

Before  detections  from  both  cameras  could  be  fused,  the  cameras  both  required 
calibration,  to  compute  the  intrinsic  and  extrinsic  parameters.  A  camera  calibration 
toolbox  from  (Bouguet,  2013)  was  used  to  solve  for  the  camera  matrices  KL ,  KR ,  the 

rotation  vector,  R ,  and  Translation  vector,  T.  Comer  detection  was  used  to  estimate  the 
fundamental  matrix.  By  taking  video  with  both  cameras  running  at  once,  the  spatial 
relationship  between  them  was  determined.  Figure  27  displays  the  extrinsic  parameters 
of  the  cameras,  and  how  they  spatially  relate  to  the  calibration  checkerboard  grids  1-10. 
The  left  and  right  cameras  existed  on  different  planes  in  all  axes,  and  registration  of  their 
images  required  knowledge  of  a  projective  transformation  matrix.  The  camera  calibration 
outputted  the  rotation  matrix,  R,  and  translation  vector,  T,  which  are  factors  of  the 
projective  geometry  between  the  two  cameras. 
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Figure  27:  The  extrinsic  parameters  of  the  two  camera  setup  is  shown.  The  checkerboards  that  were  used 
for  calibration  depicted  by  the  colored  grids  numbered  1-10.  Note  that  they  are  not  coplanar  in  any  axis. 

3.4.3  Image  Transformation  and  Registration 

If  the  point  in  space  is  known  approximately,  the  projective  transformation  can  be 
obtained  to  register  images  from  stereo  cameras.  First,  the  points  in  relation  to  the  right 
camera  and  the  target  image  plane,  xit  y„  z,  are  needed.  These  points  are  output  from  the 
function  footprint  ( )by  (Welborn,  2013).  Next,  these  points  are  output  with  an 
algorithm  from  (Mariottini  &  Prattichizzo,  2005)  shown  by  (Eq.  28)  . 

UR=[xt  yt  z,  if  (Eq.  27) 


ut 

vi 


k[rt\\t]us 


(Eq.  28) 
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Where: 


xi,yi,zi  =  Spatial  location  of  image  corners  and  (7=1 ,2,3,4)  with  respect  to  the 
right  camera  frame. 

After  the  points  corresponding  to  the  projective  transfonn  are  found,  a  transform 
was  created  with  the  MATLAB  function,  Tform=fitGeoTransform(U,  X, 

‘  Projective  ’ ) .  Where  X  is  a  4x2  matrix  from  the  new  comers  ui ,  and  v;  with 

z=[  1 ,2,3,4]  and  U  is  the  4x2  matrix  of  untransformed  corners  on  the  left  camera.  Next,  the 

transfonn  is  applied  to  an  image  with  the  MATLAB  function 

imageTf orm= imwarp ( Image , Tf orm,Rref ) . 


Figure  28:  Results  of  projective  transform  registration  using  SIFT  and  RAN  SAC.  Shows  the  stereo 
disparity  of  the  two  cameras.  RGB  is  red  image,  and  SWIR  is  represented  by  green  image.  Note  that 
disparity  is  only  minimal  at  one  location,  the  dismount,  whereas  disparity  increases  with  greater  distance. 

Since  the  cameras  were  not  perfectly  aligned  in  3-D  space,  the  distortion  changes 
for  different  target  ranges.  This  fact  can  be  accounted  for,  given  the  distance  to  the 
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dismount  is  known.  Solving  for  slant  distance,  z,  by  geometry  of  camera  position  using 
(Eq.  3),  the  projective  transformation  matrix  for  the  left  camera  can  be  determined. 

3.4.4  Epipolar  Geometry 

Since  the  rotation  vector  is  not  zero,  and  the  translation  vector  is  greater  than  zero 
in  more  than  one  plane,  the  epipolar  geometry  resembled  a  model  more  similar  to  the 
non-parallel  stereo  model.  This  model  is  still  valid  for  detennining  registration  offset 
with  two  cameras,  however  the  epipolar  lines  may  no  longer  be  orthogonal  to  the  y  axis. 
The  results  of  this  are  shown  by  the  stereo  rectified  images  from  camera  calibration  in 
Figure  29. 


Figure  29:  Left,  VIS  camera  calibration  image.  Right,  SWIR  Camera  calibration  image.  Epipolar  lines 
depicted  in  blue,  show  the  geometric  relationship  between  the  two  cameras  after  stereo  rectification. 

3.4.5  Reflectance  Conversion 

Before  applying  the  dismount  detection  algorithm,  raw  pixel  values  were 
converted  to  reflectance.  Multiple  radiometric  correction  methods  were  experimented 
with,  and  trial  and  error  was  used  to  determine  the  best  method.  Flat  field  correction  just 
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required  one  spectrally  non-reflecting  material,  such  as  a  black  calibration  panel  or 
asphalt.  ELM  corrections  perform  best  with  two  or  more  reference  materials  (which 
reflectance  is  known  for),  and  then  applying  a  linear  gain  to  the  raw  data.  However,  the 
ELM  method  was  less  desirable  for  reasons  mentioned  in  Section  2.  In  test  scenes,  white 
and  dark  panels  were  available,  and  their  signatures  are  shown  in  Figure  30.  During  test 
flights,  a  few  objects  with  contrasting  reflectance  were  in  most  frames;  concrete,  asphalt, 
runway  markings,  and  clothing  all  had  unique  spectral  signatures.  Their  reflectance 
values  were  found  with  a  spectral  library  or  the  ASD  FieldSpecPro  3.  Thus,  ELM  or  flat 
field  correction  could  then  be  used  to  account  for  environmental  illumination.  ENVI  5.0 
was  used  to  apply  correction  to  imagery  (Excelis  Inc,  2013). 


Figure  30:  Reflectance  values  for  white  and  dark  Labsphere  calibration  panels 

3.4.6  Video  Processing 

The  algorithm  for  processing  video  was  unique  to  the  SUAS  MSI  that  was 
constructed.  Since  the  filter  wheel  was  spinning  while  the  aircraft  was  moving,  the 
images  had  to  be  overlaid  to  create  an  image  cube.  The  process  is  as  follows:  A  frame 
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was  captured  with  the  filter  in  the  first  position,  \ .  Next,  the  filter  wheel  rotates  to  the 


second  position,  A2 ,  and  another  frame  is  captured,  ft  was  necessary  to  achieve  proper 
overlap  between  the  first  frame,  F\ ,  and  the  second  frame,  F2 ,  for  i  =  1,2. .  .n.  To 
spatially  align  pixels  in  F\  and  F2 ,  registration  was  done  using  SIFT  and  RANSAC  for 
point  feature  extraction  and  homography  transformation.  Figure  3 1  illustrates  the 
footprint  of  frames  F]  to  Fn  for  two  wavelengths,  A, ,  A2 .  Since  the  VIS  camera  has  no 
filter  wheel,  the  frames  were  captured  at  25  fps  for  the  entire  sequence. 


Figure  31:  Frame  capture  sequence  for  spectral  image  construction 

3.5  Analysis  Approach 

To  measure  the  effectiveness  of  the  imagery  system,  qualitative  and  quantitative 
metrics  were  used.  The  ground  based  HS1  was  compared  to  the  SUAS  MSI,  and  then 
ground  data  was  compared  to  aerial  data.  Each  metric  judged  different  aspects  of  the  data. 
ROC  curves  for  each  system  quantitatively  measure  the  effectiveness  of  the  detection 
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algorithm  and  sensor  in  detecting  skin.  IQ  metrics  judged  the  amount  of  distortion  and 
motion  blur.  Pixel  density  was  calculated  for  each  image  to  relate  each  imager  despite 
different  lens  focal  lengths.  Airframe  effectiveness  metrics  were  used  as  well,  including 
time  on  target,  gyration  rates,  and  accelerations  from  a  high  resolution  IMU.  Qualitative 
metrics  were  used  for  comparing  video  quality  between  test  flights,  and  between  imagers. 
Factors  that  were  qualitatively  analyzed  included  dismount  illumination,  overall  video 
quality,  and  visibility. 

3.5.1  ROC  Comparison  Analysis 

The  detection  systems  were  compared  with  receiver  operating  curve  (ROC), 
which  displays  classification  probability  over  a  varying  detector  threshold.  For  this 
analysis,  truth  labels  were  generated  from  the  imagery.  Next,  the  detection  output,  or 
score,  was  thresholded  for  probability  of  true  positive  rate  versus  false  positive  rate.  For 
the  NDSI  Algorithm,  a  single  score  method  was  adapted  to  generate  a  ROC  (Beisley, 
2012).  Area  under  the  curve  (AUC)  is  a  single  score  that  was  used  to  compare  each 
curve.  A  ROC  Curve  is  an  objective  metric  that  compares  the  sensor,  and  classification 
algorithm  of  two  systems. 

3.5.2  Dismount  Detection  Scenario 

To  recreate  an  operational  environment,  the  data  can  be  collected  from  an  aerial 
imager  with  real  time  video.  To  accomplish  this  task,  there  must  be  a  synergy  of 
elements.  A  human  will  be  placed  in  an  outdoor  operational  environment.  The  plane  will 
launch  from  a  remote  location,  and  an  operator  will  view  data  from  a  ground  station.  The 
approximate  coordinates  of  the  human  were  given  to  the  operator,  and  the  operator  was 
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tasked  with  identifying  the  dismount.  After  a  racetrack  pattern  was  established,  fine 
adjustments  were  made  to  the  routing  path  to  view  the  dismount. 

3.5.3  Flight  Path  Planning 

A  challenge  of  object  detection  is  collecting  enough  light  off  the  object  with  the 
sensor.  This  is  where  planning  can  be  done  to  consider  all  aspects  of  lighting,  camera 
exposure,  gain,  and  pixel  density.  A  unique  challenge  arises  when  detecting  objects  that 
are  small  and  concealed,  given  that  there  is  little  natural  light  that  arrives  from  an 
awkward  angle,  and  the  sensor  is  at  a  large  slant  range.  Autonomous  flight  for  the 
research  utilized  straight  and  level  routes;  however  wind,  lighting,  and  flight  restrictions 
dictated  the  autopilot  waypoints.  Sun  angle  was  especially  important  for  the  chosen  skin 
detection  algorithm;  patterns  were  flown  to  maximize  flight  time  with  the  sun  behind  the 
sensor.  Test  flight  telemetry  was  converted  to  a  Flat  Earth  Model  coordinate  system  in 
order  to  produce  6  DOF  data.  MATLAB  code  for  extracting  the  flat  earth  model  from  the 
autopilot  is  shown  in  the  Appendix.  This  allowed  the  flight  trajectory,  and  relative 
position  of  the  airframe  to  the  ground  aim  point  to  be  computed.  Test  flights  recorded 
high  resolution  attitude  and  IMU  data.  Gyrometers  measured  angular  rates,  and 
accelerometers  measured  acceleration  in  the  three  aircraft  body  frame  axes  up  to  50Hz 
sampling  frequency.  The  airframe  accelerations  and  rotations  were  compared  with 
imagery  metrics. 

3.5.4  Imagery  Metrics  Applied 

Decision  making  ability  was  judged  by  multiple  objective  metrics.  Metrics  were 
applied  to  the  fused  imagery  after  post  processing  the  data.  These  objective  metrics  are 
foundational  for  comparing  the  tradeoffs  of  the  ground  based  system  with  the  SUAS  MSI 
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system.  The  metrics  in  Table  1  are  known  to  measure  image  and  data  fusion  quality.  The 
metrics  were  applied  with  MATLAB  software  for  a  single  frame  of  video. 


3.6  Summary 

This  chapter  covered  the  experimental  procedure  and  methodology.  It  explained 
the  aerial  testing  platform,  sensor  type  and  imaging  apparatus,  measurements  for 
quantifying  sensor  effectiveness,  and  comparison  technique  to  answer  the  research 
questions. 
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4.  Analysis  and  Results 


4.1  SUAS  MSI  Ground  Testing  and  Tuning 

There  will  always  exist  time,  budget,  and  testing  availability  constraints  that  limit 
the  scope  of  the  research.  For  the  SUAS  MSI,  a  few  issues  could  have  been  avoided. 

Ground  tests  revealed  that  focus  was  poor  at  long  range  in  one  wavelength. 

LoCA,  as  mentioned  in  section  2,  was  most  likely  the  cause  for  poor  focus  with  one 
wavelength.  LoCA  required  the  camera  to  utilize  a  mean  focus  setting  between  1050  and 
1550nm  bandpass  filters.  Figure  32  depicts  a  mean  focus  setting  for  both  bands  to 
minimize  LoCA.  Vignette  is  visible  in  most  scenes,  and  characterized  by  low  pixel 
values  around  the  edges  of  the  image  center.  Diagrams  of  different  aberrations  and 
vignette  are  located  in  the  Appendix.  Light  bleeding  was  another  issue  that  occurred 
from  gaps  in  the  optical  system;  an  example  is  a  gap  between  the  lenses  and  filters.  The 
plastic  filter  housing  without  a  proper  non-reflective  coating  also  caused  light  bleed. 
Nunez  (2009)  sealed  the  gap  between  lens  and  filter,  and  this  was  done  on  the  test  system 
as  well  to  block  unwanted  light  during  later  tests.  The  filter  arrangement  perfonned  well, 
and  a  relatively  constant  rate  spin  with  filter  wheel  was  achieved  by  using  fresh  batteries 
for  the  motor  driver.  Digital  artifacts  were  present  in  the  video.  Sato  (2006),  notes  that 
digital  artifacts  are  often  created  by  the  Analog  to  Digital  Convertor  (ADC)  if  sampled  at 
too  low  of  a  rate,  and  they  can  be  seen  in  smooth  tonal  areas. 
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Figure  32:  Left,  1050nm  image.  Right,  1550nm  image.  Focus  is  at  a  mean  setting.  Note  the  car  in 
background  between  the  filter  change  (Filtering  speed  was  0.50  seconds/filter  change). 

Applying  the  dNDsi  algorithm  required  scaling  and  registration  of  the  image 
planes.  Since  the  cameras  were  mounted  separately,  and  they  were  of  different 
resolutions,  the  registration  had  to  account  for  the  intrinsic  and  extrinsic  camera 
properties. 

4.2  Ground  Imager  Comparison 

4.2.1  Line  Scan  HSI  Ground  Test 

The  SpecTIR  HSI  uses  digital  frame  grabber  to  store  12  bit  information  for  each 
band  (SpecIM,  2013).  This  improves  upon  the  8  bit  compression  from  the  ADC  used  by 
the  SUAS  MSI.  The  SpecTIR  benefits  from  a  simple  registration  alignment  and  closer  to 
a  parallel  stereo  arrangement  as  displayed  in  Figure  6;  the  only  offset  in  the  x  axis  is 
required  for  registration  of  the  VIS  and  SWIR  bands.  Results  from  the  dNDsi  and  MF  in 
follow  in  Figure  33  through  Figure  35  for  the  SpecTIR  MSI.  Figure  36  through  Figure 
38  show  results  from  the  SUAS  MSI.  The  same  bands  were  used  for  both  imagers  for  a 
relevant  comparison. 
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Figure  33:  SpecTIR  10m  range  image.  Left,  dNDSI  detection  score.  Right,  dNDSI  with  tl 


Figure  34:  SpecTIR  10m  range  image.  Left,  MF  detection  algorithm  score.  Right,  MF  wil 
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Figure  35:  SpecTIR  ROC  Curves  for  two  skin  detection  algorithms.  ROC  for  dNDsi  is  plotted  as  blue.  The 
matched  filter  only  using  4  out  of  363  available  bands  produced  the  result  in  red.  These  detections 
produced  AUCdNDSi  =0.9694,  AUCMf=0.9478. 


Figure  36:  SUAS  MSI  detections.  Left,  d^osi  score,  dj^si  with  threshold. 
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Figure  37:  SUAS  MSI  Camera  ROC  Curves  for  two  skin  detection  algorithms.  ROC  for  dNDSI  is  plotted  as 
blue.  ROC  for  MF  using  4  bands  is  plotted  in  red.  These  detections  produced  AUCdNDsi  =0.8912, 

AUCmf=0.9052. 


Figure  38:  SUAS  MSI  MF  detection  with  binary  threshold  applied 
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The  ground  test  dismount  scenario  compared  the  ground  based  HSI  and  SUAS 
MSI  at  similar  GSD.  The  NIIRS  score  was  calculated  using  the  method  described  in 
section  2  and  similar  values  were  obtained  for  each  imager  due  to  the  large  weighting  of 
GSD.  The  CPBD  score  was  near  1.00  for  both  which  meant  that  blur  was  minimal,  and 
this  was  logically  valid  for  stationary  imagers  and  dismounts.  For  skin  detections,  the 
HSI  consistently  obtained  greater  AUC  values  compared  to  the  SUAS  MSI.  As 
mentioned  in  the  research,  this  may  be  attributed  to  registration  errors,  optical 
transmission,  aberrations,  digital  resolution,  or  exposure  settings.  With  all  these  issues 
considered,  the  SUAS  MSI  performed  well  given  its  significantly  lower  cost.  A 
summary  of  the  skin  detection  platforms’  parameters  and  results  are  given  in 
Table  4. 


Table  4:  Sensor  Comparison 


Parameter 

Ground  Based  HSI 

SUAS  MSI 

Specifications 

Focal  Length 

18.5-22.5  mm 

35  mm 

FOV(vertical) 

32° 

26° 

Spectral  Bands 

3634 

4 

Design 

Line  Scanning 

Filter  Wheel 

Range  (5.0  cm  pixel) 

50  m 

71  m 

Frame  Collection  Rate 

60  Hz 

0.5  Hz 

Digital  Resolution 

12-bit 

8-bit 

Mass 

>15kg 

1kg 

Performance 

NIIRS  /  GIQE  (  60  m) 

8.15 

8.70  (PD  =  6.53%) 5 

CPBD 

0.983 

0.999  (PD  =1.75%) 

dNDsi  AUC6 

0.974 

0.912  (PD  =  6.55%)* 

MF  AUC7 

0.972 

0.937  (PD  =  3.67%)* 

4  Only  the  same  4  bands  as  the  SUAS  MSI  were  used  for  detections 

5  PD:  Percent  (%)  Difference,  *  indicates  statistical  significance  (p-value  <  0.05) 

6  Averaged  over  a  sample  size  of  n=3  images,  SUAS  MSI:  OdNDsi  =  0.0200,  Ground:  OdNDsi  =  0.0143 

7  Averaged  over  a  sample  size  of  n=3  images,  SUAS  MSI:  oMf  =  0.0208,  Ground:  oMF  =  0.0245 


77 


4.2.2  Lighting  Angular  Effects 

Lighting  was  predicted  to  play  an  important  role  in  detection  ability.  Adequate 
lighting  was  maintained  in  most  tests,  and  camera  gain  and  exposure  settings  were 
adjusted  to  optimize  IQ.  One  can  refer  to  Figure  11,  where  a  was  defined  as  sun  zenith, 
9i  was  defined  as  incident  angle  of  the  dismount,  and  6d  was  defined  as  the  viewing 
angle  in  relation  to  the  dismount  surface  normal.  Different  lighting  angles,  0j,0d  were 

tested  at  a  constant  a  to  characterize  the  effect  of  lighting  angle.  The  dNDSi  classification 
algorithm  was  used  for  all  detections.  Ground  testing  results  from  ROC  inspection  and 
AUC  indicate  no  significant  impact  due  to  lighting  angles.  However,  the  ground  testing 
environment  was  subject  to  less  lighting  extremes  than  the  aerial  enviromnent. 
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Table  5:  Lighting  angle  effect  on  detections 


False  Positive  Rate 


4.2.3  SUAS  MSI  Video  Processing 

The  initial  settings  were  maintained,  and  the  filter  was  set  to  change  at  equal 
intervals.  An  image  sequence  was  filmed  of  a  moving  dismount,  and  a  Simulink  program 
processed  footage  and  applied  the  NDSI  algorithm.  The  processing  program 
automatically  used  the  rotation  rate  of  the  filter  wheel  to  trigger  frames  when  the  filter 
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wheel  was  positioned  correctly.  Figure  39  shows  the  moving  arms  of  the  dismount,  and 
how  the  algorithm  did  not  update  until  the  next  filter  exchange.  Therefore,  the  slow  filter 
rate  equates  to  a  decrease  in  viewing  rate.  Compared  to  the  AFIT  monocular  system,  this 
SUAS  MSI  system  displays  near  the  same  frames  per  second,  however,  it  performs  best 
with  a  stationary  target.  This  is  because  frames  from  each  wavelength  are  acquired  in 
succession,  rather  than  at  the  same  instant.  Another  difference  with  the  monocular 
system,  which  utilized  two  cameras,  was  the  amount  of  aberration.  Each  camera  on  the 
monocular  system  had  separate  filters  for  1050nm,  and  1550nm,  could  be  adjusted 
separately,  and  had  a  fore  optic  with  maximum  aperture;  this  allowed  for  less  vignette, 
chromatic,  and  spherical  aberration  in  images.  More  images  of  the  optics  effects  are 
included  in  the  Appendix. 


Figure  39:  Video  Sequence  after  post  processing  and  NDSI  algorithm  applied.  Filter  rate  was  1.00  sec/per 
filter  change.  Test  subject  is  waving  arms  and  false  detection  is  visible  when  arms  move.  Vegetation  is 
visible  due  to  water  absorption  characteristics  shared  with  skin. 
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4.2.4  Flight  Test  Results 


Figure  40:  Sig  Rascal  1 10  in  flying  configuration  with  MSI  sensor  payload 

Flight  testing  was  performed  to  determine  MOEs  for  the  operational 
configuration.  Aerial  Imagery  collection  for  the  SUAS  MSI  revealed  many  challenges 
with  dismount  detection,  but  it  also  provided  an  environment  to  test  possible  solutions. 
Camera  parameters  including  auto-gain  skewed  pixel  values  could  not  be  used  with  the 
SUAS  MSI  system.  For  the  SWIR  camera,  the  auto-gain  recalculated  at  a  maximum 
10Hz,  and  successive  filter  changes  caused  auto-gain  to  change  Operational  (OPR) 
settings  on  the  camera.  Each  OPR  setting  had  a  unique  gain,  exposure,  and  gamma  level 
that  was  optimized  for  a  specific  level  of  environment  illumination.  This  resulted  in  over 
saturated,  or  under  saturated  images  when  filters  were  alternated.  Therefore,  this  setting 
had  to  be  deactivated  for  proper  data  acquisition.  Longer  exposure  times  were  required 
during  the  morning,  and  shorter  times  were  required  midday.  However,  changing  these 
settings  without  auto-gain  was  inconvenient  during  testing. 
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Setting  camera  parameters  on  the  SWIR  is  accomplished  with  EIA-232  serial 
port,  and  testing  was  done  to  establish  wireless  communication  with  the  microcontroller. 
In  flight  testing,  it  also  posed  a  challenged  to  achieve  the  proper  setting.  The  image 
sensor  may  come  in  direct  line  of  sight  with  the  sun.  The  camera  had  a  failsafe  to  cut 
power  when  the  scene  becomes  too  saturated,  and  this  protects  the  camera  in  direct 
sunlight.  Nevertheless,  this  would  pose  a  problem  for  any  aerial  detection  scenario  that 
has  varying  sky  conditions.  Testing  showed  that  midday  detections  were  higher  quality; 
the  solar  zenith  angle  was  smallest,  and  environment  illumination  was  more  uniform. 
Detections  during  peak  zenith  were  better  with  lower  OPR  settings;  however,  later  in  the 
day,  higher  OPR  settings  were  used  to  compensate  for  uneven  ground  illumination.  Real 
time  imagery  collection  requires  the  ability  to  change  camera  parameters  “on  the  fly”.  On 
the  aerial  detection  system,  it  was  observed  that  overexposure  and  underexposure  in  a 
moving  environment  significantly  decreased  IQ. 

Reflectance  conversion  for  flight  test  video  was  done  using  Flat  Field  correction 
with  good  accuracy.  If  this  method  cannot  be  performed,  illumination  invariant 
reflectance  conversion  methods  have  been  the  best  option  for  the  SUAS  MSI.  For  real 
time  processing  of  imagery,  RT  code  methods  would  be  optimal  and  not  require 
calibration  panels  for  Flat  Field  or  ELM  (Beisley,  2012).  These  methods  depend  on  the 
solar  zenith,  angle  vector  of  the  camera  with  respect  to  the  sun,  time  of  day,  and 
atmospheric  conditions  all  which  are  widely  available. 

Long  slant  range,  and  high  illumination  environments  require  additional  optical 
considerations.  Lens  selection  resulted  in  the  proper  focal  distance,  but  improper  sensor 
format  for  the  SWIR.  For  aerial  application,  size  and  weight  was  key;  for  this  reason,  the 
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fixed  focal  length  lens  was  used.  This  lens  was  significantly  smaller  than  the  OEM 
Nikon  50mm  f/1.4  F-mount  lens;  moreover,  the  sensor  size  supported  by  the  compact 
lens  was  not  well  matched  to  the  Goodrich  SWIR.  Nevertheless,  vignette  would  still 
occur  due  to  the  bandpass  filter  diameters  that  were  available.  For  these  reasons,  Region 
of  Interest  (ROI)  windowing  of  the  images  was  required  for  video.  An  auto  adjusting 
aperture,  and  optics  designed  for  the  25 jim  focal  plane  array  would  increase  sharpness 
and  reduce  distortion  but  they  still  would  not  focus  images  for  both  bands;  this  can  be 
attributed  to  aberrations  such  as  LoCA  in  the  refractive  fixed  focal  length  lens.  The  best 
solution  would  utilize  a  reflecting  lens,  because  a  reflecting  optic  is  indifferent  to 
refraction  index  of  the  material. 

4.2.5  Waypoint  Navigation 


Long  distance  from  base  (m)  +N  Long  distance  from  base  (m)  +N 

Figure  41 :  Left,  August  flight  test  pattern  for  initial  gain  tuning.  Right,  October  Flight  Test  Pattern  for 

Imagery  Collection 

Weather  conditions  affected  test  results  and  significantly  more  wind  was 
encountered  during  imager  collection  than  gain  tuning.  These  conditions  accounted  for 
deviations  from  the  intended  waypoint  locations.  The  deviations  of  the  test  flight  pattern 
can  be  compared  to  the  gain  tuning  test  pattern  with  Figure  41. 
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Battery  power  was  considered  as  a  limiting  factor  for  flight  time.  Having 
sufficient  battery  power  for  the  operational  demo  was  dependent  on  wind  speed  and 
battery  degradation.  Flight  tests  were  scheduled  for  15  minutes  duration,  and  could  be 
increased  by  adding  larger  batteries.  These  limits  are  less  than  the  1  hour  flight  time 
proposed  in  the  conceptual  system  design.  This  time  limitation  may  impact  dismount 
detection  in  remote  areas  that  do  not  have  access  to  chargers,  or  spare  batteries.  The  flight 
time  limits  were  likely  reached  earlier  than  anticipated  due  to  high  winds  and  required 
further  planning  to  optimize  testing  time. 

4.2.6  Flight  Video  Processing 

Faster  filtering  rates  were  needed  to  minimize  stereoscopic  effect  of  images. 
Having  two  images  taken  from  a  large  distance  apart,  creates  a  stereo  vision  effect  on  3-D 
objects.  This  effect  was  noticed  with  registration  of  images,  and  it  hindered  registration 
performance  of  the  first  test  flights  with  1.0  fps.  The  next  flight  test  increased  the 
filtering  rate  to  2.0  fps,  and  this  decreased  the  time  between  images.  It  improved 
registration  performance,  and  it  also  decreased  effects  of  the  resultant  wind.  Figure  42 
shows  a  sample  flight  pattern  used  for  data  collection.  Figure  43  shows  variations  in 
working  distance  and  pixel  size  of  the  MSI  imager  for  a  segment  of  the  flight  pattern. 
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Figure  42:  A  sample  flight  pattern  is  shown  in  blue  on  both  images.  Left,  the  sensor  aimpoint  is  shown  in 
green.  Right,  the  sensor  FOV  is  projected  on  the  ground  is  shown  in  red. 
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Figure  43:  The  sample  pattern  is  analyzed.  Left,  variation  in  sensor  working  distance  is  plotted  as  a 
function  of  time.  Right,  pixel  size  is  shown  as  a  function  of  time. 


Feature  detection  with  SIFT  and  RANSAC  was  applied  to  the  imagery  to  register 
frames.  Using  feature  detection  technique  with  SWIR  images  alone  was  difficult.  This  is 
because  each  successive  frame  displayed  data  from  a  different  wavelength.  Furthermore, 
feature  detection  with  vision  based  methods  alone  was  difficult  for  areas  such  as  a 
runway  or  forest,  with  very  few  intensity  based  landmarks.  Even  at  the  2  fps  filtering 
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rate,  there  was  usually  significant  movement  between  each  frame,  which  caused  some 
projective  transforms  to  stretch  to  near  infinity.  Limiting  the  video  data  collection 
window  to  straight  line  flight  paths  reduced  most  of  the  invalid  transforms.  A  3-D  view 
of  sensor  aim  geolocation  results  using  autopilot  telemetry  is  depicted  in  Figure  44.  The 
resulting  projective  transform  of  a  video  sequence  using  autopilot  telemetry  is  depicted  in 
Figure  46. 


Figure  44:  Test  Data  with  sensor  aim  vector  plotted  as  blue  dotted  lines 
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Figure  45:  Test  Data  with  projective  transform  plotted  as  red  lines 
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Figure  46:  Projective  transformation  of  a  video  sequence  from  autopilot  telemetry 
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The  VIS  camera,  collecting  at  30  fps,  showed  significant  interlacing  blur  due  to 
motion  and  vibrations  from  the  motor,  and  wind  in  the  air.  Figure  47  quantifies  the 
motion  blur  with  the  CPBD  IQ  metric  for  the  upwind  and  downwind  portions  of  flight 
video.  Higher  CPBD  occurred  during  the  steady  upwind  portion  of  the  pattern,  while 
entering  turn  and  downwind  decreased  the  CPBD.  Turns  and  downwind  flight  caused 
higher  movement  rates  in  the  longitudinal  axis  (roll)  and  normal  axis  (yaw)  relative  to  the 
aircraft  body  frame.  Figure  48  shows  average  angular  movement  rates  for  each  portion 
of  the  flight  pattern.  Low  CPBD  was  indicated  by  blur  and  distortion  in  the  imagery. 

Blur  interfered  with  image  registration  for  the  vision  based  methods. 


Figure  47:  Cumulative  Probability  of  Blur  Detection  (CPBD)  is  plotted  over  time  for  upwind  and 
downwind  portions  of  the  flight  pattern.  The  upwind  run  had  a  higher  overall  CPBD,  and  averaged  71.6%. 
Downwind  run  averaged  66.5%  CPBD  value.  A  larger  %  value  of  CPBD  indicates  less  blur. 
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Figure  48:  Comparison  of  the  test  patterns  roll,  pitch,  and  yaw  rates  averaged  for  10  upwind  and  10 

downwind  runs. 


Figure  49:  Background  images  from  flight  data.  Left,  1050nm.  Middle,  1550nm.  Right,  VIS. 


Vibrations  greatly  affected  video  quality.  Video  collection  from  the  SUAS  proved 
effective,  but  faced  blur  and  distortion  during  wind  turbulence  and  high  throttle 
maneuvers.  Figure  49  shows  significant  blur  on  SUAS  MSI  imagery,  and  this  is 
indicated  by  waves  most  visible  in  the  right,  colored  image.  Feature  tracking 
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performance  was  degraded  with  the  blur  but  the  SIFT  and  RANSAC  still  proved  robust  in 
image  registration. 


Figure  50:  SWIR  test  flight  imagery  with  feature  point  matching  applied  for  projective  transformations  and 
image  registration.  Top  left,  first  1050nm  frame  with  feature  matches  indicated  by  the  colored  points.  Top 
right,  original  image  from  next  1050nm  frame.  Bottom  left,  first  frame  with  projective  transformation 
applied.  Bottom  right,  second  frame  with  projective  transformation  applied. 


Projective  transform  calculation  with  attitude  data  was  possible  with  methods 
discussed  in  section  2,  however,  application  of  the  algorithms  in  MATLAB  were 
inefficient.  A  C++  based  platform  could  utilize  attitude  and  vision  feature  detection 
joined  by  an  Extended  Kalman  Filter  (EKF)  to  estimate  a  projective  transform  more 
efficiently  (Beard  &  McLain,  2012).  Feature  detection  with  moving  imagery  is  quick  and 
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efficient  with  the  SIFT  and  RANSAC  methods  discussed  in  section  3.  Figure  50  shows 
results  of  feature  processing  for  a  video  sequence  in  one  wavelength.  Calculation  of  the 


projective  transformations,  and  registration  was  done  in  MATLAB.  The  process  was 
repeated  for  each  wavelength  to  assemble  an  image  cube.  Figure  5 1  through  Figure  54 
show  the  step  by  step  process  for  registering  and  assembling  an  image  cube. 


Figure  5 1 :  Left,  1050nm  video  frame,  after  projective  transformations  applied.  Right,  1550nm  video  frames 


91 


Figure  52:  Point  matching  between  the  1050nm  and  1550nm  image  frames,  before  final  image  cube. 


Figure  53:  Registered  video  sequence  of  1050nm  using  SIFT,  RANSAC,  and  homography  transformation 
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Figure  54:  Left,  1050nm  image.  Right,  1550nm  image 


Figure  55:  Left,  NDSI  algorithm  applied  to  the  aerial  image  sequence.  Right,  threshold  image,  where  white 

area  denotes  vegetation  or  skin 
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The  NDSI  algorithm  also  is  able  to  detect  vegetation.  Figure  55  shows  an  aerial 
scene  with  the  detection  algorithm  applied.  Using  more  bands  can  reduce  false  alarm  on 
unwanted  material.  Robust  skin  detection  involves  applying  a  rules  based  algorithm  with 
the  NDSI  and  NDGRI  to  achieve  detection.  However,  with  these  results,  we  infer  that 
such  an  algorithm  can  detect  skin  with  a  relatively  high  probability. 


\ 


False  Positive  Rate 


Figure  56:  SUAS  MSI  fight  test  detection  example.  Left,  MF  score  from  VIS  Bands.  Middle,  MF  score 


thresholded.  Right,  ROC  Curve  with  resulting  AUC  =  0.91 18 


Flight  test  detections  were  on  average  less  accurate  than  the  ground  test  results, 
due  to  focus,  vibration,  and  registration  error.  However,  multiple  detections  gave  high 
AUC  values.  This  may  be  attributed  to  the  low  background  noise  from  an  airstrip  scene, 
because  there  were  less  buildings  and  obstructions.  Figure  56  shows  a  detection  result 
with  the  MF  at  approximately  20m  slant  distance.  The  results  prove  that  aerial  dismount 
detection  is  feasible  with  the  proposed  equipment. 

4.3  Chapter  Overview 

This  chapter  gave  the  results  from  ground  testing,  flight  testing,  and  analysis  of 
data  for  this  thesis.  The  flight  expedient  prototype  system  was  compared  to  a  proven 
ground  based  imager,  and  issues  faced  for  the  MSI  onboard  an  SUAS  were  revealed  from 
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flight  testing.  Proposed  refinements  to  the  prototype  requirements  were  explained  for  the 


system. 


4.4  Investigative  Questions  Answered 


1 .  Is  MSI  effective  for  dismount  detection  on  an  SUAS? 

•  The  literature  and  results  explain  how  aerial  imagery  can  improve  the 
ability  to  distinguish  skin  from  background. 

•  It  was  demonstrated  that  aerial  dismount  detection  exhibited  AUC 
values  as  high  as  0.91,  which  indicates  a  reasonable  effectiveness. 

2.  What  type  of  MSI  system  is  needed  for  an  aerial  platform? 

•  The  original  operation  requirements  outlined  a  possible  solution  for 
dismount  detection  from  an  SUAS;  however,  refinements  to  the 
proposed  solution  were  noted  including: 

o  High  frame  rate  by  using  faster  filter  change  speeds  or  a 
patterned  filter  array; 

o  Refractive  optics; 

o  Automatic  gain  and  exposure  adjustment; 

o  Kalman  filtering  for  image  registration. 

•  Improvements  to  the  design  would  increase  the  utility  of  the  system 
for  minimal  additional  cost. 

3.  Which  camera  parameters  make  aerial  dismount  detection  a  challenge? 

•  There  were  problems  encountered  due  to  sun  angle.  This  was  noted 
during  flight  testing,  and  waypoint  planning  considered  the  location  at 
of  sunlight  in  relation  to  the  MSI  sensor. 

•  Modifying  gains  in  the  field  was  cumbersome,  and  wireless  serial 
communication  to  the  camera  system  would  improve  detection 
effectiveness. 

•  A  fixed  body  sensor  was  chosen  for  cost  and  weight,  but  a  gimbaled 
sensor,  even  in  one  axis,  would  be  useful  for  improving  IQ. 


95 


•  Frame  capture  speed  affected  image  blur,  feature  matching,  and 
registration.  Having  a  slightly  higher  frame  rate,  by  increasing 
fdtering  speed,  provide  some  improvement  during  testing. 

4.  What  metrics  are  best  to  compare  dismount  detection  ability? 

•  NIIRS  values  were  higher  on  the  SUAS  MSI  by  a  percent  difference  of 
6.53  %.  However,  NIIRS  was  largely  a  function  of  GSD  or  pixel  size 
rather  than  spectral  properties. 

•  On  the  ground,  CPBD  exhibited  a  higher  perfonnance  by  the  SUAS 
MSI  percent  difference  of  1.745  %  between  imagers.  The  small 
difference  in  CPBD  on  the  ground  simply  validated  the  assumption 
that  the  imagers  exhibited  negligible  blur  on  the  ground.  In  the  air, 
CPBD  varied  as  high  as  33.41  %  compared  to  non-moving  imagery  for 
a  pattern  speed  of  12m/s  and  altitude  of  100ft. 

•  ROC  AUC  values  were  higher  on  the  ground  based  imager  by  6.553% 
for  the  disjDsi  detector,  and  3.672%  for  the  MF  detector.  The  difference 
in  detection  capability  was  small  using  subjective  analysis.  However, 
AUC  Values  were  higher  for  the  ground  based  imager  compared  to  the 
SUAS  MSI  with  statistical  significance  (p  <  0.05). 
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5.  Conclusions  and  Recommendations 


5.1  Chapter  Overview 

This  chapter  outlines  the  conclusions  of  the  research.  It  explains  the  difficulties 
associated  with  skin  detection  on  an  SUAS  with  the  proposed  MSI  platform,  and  reasons 
for  those  difficulties.  Also,  it  recommends  avenues  for  future  research. 

5.2  Conclusions  of  Research 

The  literature  and  results  both  show  that  Spectral  imaging  with  a  SUAS  is 
difficult  due  to  cost,  computation,  communications,  and  environmental  forces  on  small 
sized  airframes.  Nevertheless,  the  SUAS  MSI  exhibited  dismount  detection  performance 
comparable  to  a  larger,  more  capable  sensor  with  only  a  5. 1 12  %  difference  for  ROC 
AUC  values.  Refinements  to  the  system  can  certainly  improve  its  ability  to  capture, 
transmit,  process  data,  and  yield  valuable  results  for  users. 

5.2.1  Proposed  Modifications 

The  proposed  solution  had  a  few  issues  that  hindered  performance,  but  could  have 
been  avoided.  The  first  was  using  analog  output  from  the  SWIR  to  capture  imagery.  The 
NTSC  analog  to  digital  conversion  process  limited  video  resolution  from  5 12  to  480 
vertical  pixels,  decreased  the  sampling  resolution  from  12  bit  to  8-bits,  and  recorded 
frames  between  filter  changes.  Upgrading  the  system  to  a  digital  frame  grabber  would 
increase  data  quality.  The  second  modification  addresses  communications  for  video  data. 
Analog  transmission  did  not  retain  the  same  level  of  IQ  as  the  video  stored  on  board 
because  static  and  noise.  Video  on  the  ground  station  was  only  usable  when  the  SUAS 
was  line  of  sight.  Digital  may  enable  higher  resolution  data  transmission.  The  third 
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modification  addresses  lens  selection.  The  lens  selection  affected  the  quality  of  imagery 
from  both  SWIR  bands.  Due  to  the  separation  in  wavelength  between  1050nm  and 
1550nm,  a  non-refractive  lens  would  eliminate  longitudinal  chromatic  aberrations.  Also, 
reflective  lenses  are  more  cost  effective  than  refractive  lenses  made  designed  for  the 
SWIR  bands.  The  forth  modification  addresses  the  filtering  system.  The  current 
equipment  can  operate  with  a  filter  speed  and  frame  trigger  at  30Hz.  This  would 
effectively  allow  more  overlap  between  frames.  3-D  spatial  transfonnation  from  both 
attitude  and  vision  processing  would  have  also  been  helpful  for  accurate  registration. 
Leveraging  both  highly  accurate  autopilot  capabilities  and  fast  feature  detection 
capabilities  of  computer  vision  with  Kalman  filtering  provides  opportunity  for  higher 
probability  of  detection.  All  of  these  modifications  would  help  improve  dismount 
detections  and  make  an  SUAS  MSI  system  practical  for  deployment. 

5.2.2  Flexible  Design 

The  proposed  system  has  the  ability  to  offer  SUAS  operators  a  flexible  design  for 
dismount  detection.  A  flexible  design  would  allow  the  system  to  operate  across  multiple 
environments  with  small  modifications.  For  example,  an  airframe  capable  of  SAR  in 
austere  environments  such  as  thunderstorms  forest  fires,  or  high  nuclear  radiation  zones 
would  decrease  the  likelihood  that  a  failure  would  occur  during  SAR  operations.  A 
ruggedized  platfonn  that  can  adapt  to  different  environments  would  increase  the 
capabilities  of  the  system. 

Incorporating  the  spectral  imager  also  adds  to  the  flexible  design.  For  example, 
the  proposed  solution  can  be  adapted  for  any  type  of  object  detection.  Depending  upon 
which  material  must  be  identified,  the  filters  can  be  exchanged  for  bandpass  filter  with 
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different  wavelengths  or  transmission  characteristics.  Furthermore,  more  spectral 
detection  algorithms  can  be  applied  to  the  imagery  on  the  ground  station  to  increase  the 
detection  accuracy  for  other  materials.  The  ability  to  use  more  spectral  bands  and 
detection  algorithms  can  increase  the  utility  of  the  SUAS  MSI  for  multiple  scenarios. 

Despite  the  current  setbacks,  spectral  imaging  is  still  perhaps  the  best  method  for 
identifying  dismounts.  The  ground  tests  from  the  SpecTIR  LISA  demonstrate  the 
accuracy  with  which  a  dismount  can  be  accurately  detected  with  only  4  out  of  363 
available  bands.  Research  by  Beisley  (2012)  has  proven  that  skin  detection  is  augmented 
by  many  spectral  wavelengths  that  were  not  used  in  this  thesis.  Different  wavelengths  or 
detection  algorithms  may  be  required  depending  on  water  content  in  the  air  or  on  the 
dismount,  and  concurrent  research  analyzed  the  perfonnance  of  skin  detection  algorithms 
in  an  aquatic  environment  (Chan,  2014).  The  proposed  system  presents  a  flexible  design 
to  detect  dismounts  with  relatively  inexpensive  equipment,  and  performance  comparable 
to  more  expensive  imagers. 

5.3  Significance  of  Research 

SUAS  fit  a  unique  niche  for  HSI  platfonns  that  can  be  useful  for  object 
identification.  Although  dismount  detection  on  an  SUAS  may  be  difficult,  this  research 
proves  that  the  proper  system  can  make  it  possible.  This  research  first  formed  a 
comprehensive  literature  review.  It  discussed  the  factors  related  to  SUAS  and  aerial 
imaging.  Next,  it  explored  the  factors  required  in  a  dismount  detection  system.  Then, 
processing  algorithms  and  sensor  fusion  techniques  were  reviewed.  The  system  was  then 
constructed  from  bare  components,  tested,  and  refined.  Furthermore,  difficulties  of  skin 
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detection  were  documented  for  future  research  or  exploration.  Finally,  assessment  of  the 
prototype  platform  was  conducted  with  objective  metrics,  and  overall  system 
effectiveness  for  dismount  detection  was  determined. 


5.4  Recommendations  for  Action 

With  this  research,  the  reader  should  identify  the  three  main  findings: 

•  SUAS  are  valuable  sensor  platforms  and  provide  valuable  information 
to  the  user  at  a  reasonable  cost.  Low  cost  systems  should  be  pursued 
for  widely  available  detection  platforms. 

•  SUAS  MSI  systems  should  be  considered  that  utilize  multiple  bands 
from  the  electromagnetic  spectrums  to  allow  the  sensor  platform  to  be 
used  for  multiple  roles  or  environments. 

•  Development  of  an  embedded  dismount  detection  platform  should  be 
pursued  to  decrease  processing  time.  Based  on  sensor  design  from  this 
research,  using  more  capable  sensors  will  yield  even  better  results. 

5.5  Recommendations  for  Future  Research 


This  research  set  the  groundwork  for  aerial  skin  detection  with  SUAS;  however, 
further  research  can  improve  the  technology  readiness  level  of  a  detection  system. 
Specifically,  these  avenues  include: 

•  Applications  of  the  SUAS  dismount  detection  system  to  waypoint 
routing  system  and  autonomous  decision  making; 

•  Applying  new  video  technologies  to  SUAS  such  as  micro-scale  HSI 
that  utilize  tunable  or  patterned  filter  arrays; 

•  Real  time  detection  processing; 

•  Pursuing  cost  effective  digital  video  recording  and  transmission; 

•  Onboard  SUAS  detection  processing  can  be  accomplished  with 
embedded  microcontrollers. 
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5.6  Summary 

The  SUAS  Detection  System  project  was  conceived  with  the  following 
objectives: 

•  Small  package  form  factor; 

•  Low  cost  development  and  deployment; 

•  Effective  and  efficient  dismount  detection. 

The  project  scope  covered  the  entire  development  process  for  an  airborne 
dismount  detector.  The  process  included  researching  available  airframes,  camera  and  lens 
technologies,  data  acquisition  and  analysis  protocols;  designing  and  constructing  package 
components,  deploying  and  operating  the  system,  parsing  data  and  distilling  results  and 
recommending  system  modifications  and  improvements. 

The  final  mountable  package  was  approximately  15  x  10  x  10cm  and  less  than  1.5 
kg  to  satisfy  the  size  objective.  Total  cost  of  the  system  was  more  than  1000%  less 
expensive  than  ground  based  systems.  This  fell  well  within  reasonable  limits  for  cost 
design  goals  for  the  system.  While  highly  dependent  of  environmental  conditions,  the 
system  was  able  to  achieve  reliable  and  positive  dismount  detection  in  each  of  the  tests 
performed.  Ease  of  use  remains  a  challenge  but  is  easily  improved  given  repetition  and 
refinement  of  processes.  In  summary  the  SUAS  Detection  System  project  met  all  design 
objectives,  and  based  on  suggested  improvements  and  refinements  should  be  considered  a 
candidate  for  further  research,  development  and  use  as  a  valuable  tool  in  visual  or  ATR 
dismount  detection  protocols. 
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6.  Appendix 

6.1  Appendix  D:  Optical  Aberrations  and  Distortion 

Optical  effects  that  caused  image  distortion  were  encountered,  including: 

6. 1. 1  Chromatic  Aberration 

Chromatic  aberration  affects  an  image  that  includes  different  wavelengths.  It  is  a 
result  of  the  differences  in  refraction  through  the  lens  for  different  wavelengths  of  light. 
A  diagram  of  the  path  of  light  is  shown  in  Figure  57.  While  one  wavelength  may  be  in 
focus  for  a  focal  distance,  another  wavelength  may  require  a  larger  focal  distance.  This 
aberration  results  in  one  wavelength  being  out  of  focus.  The  effect  is  depicted  on  Figure 
58  with  images  collected  from  different  wavelengths,  the  same  focal  distance,  and  the 
same  refractive  fore  optic  constructed  from  fused  silica. 


Figure  57:  Diagram  of  chromatic  aberration  (Mansurov,  2011) 
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Figure  58:  Ground  test  images.  Left  is  1050nm  in  focus,  Right  is  1550nm  out  of  focus. 

6.1.2  Spherical  aberration 

Spherical  aberration  affects  the  light  intensity  at  the  focal  point.  A  diagram  of 
this  type  of  aberration  is  shown  on  Figure  59.  The  spherical  distortion  due  to  improper 
lens  or  too  little  aperture  may  be  caused  from  excess  light  at  the  image  center,  or 
oversaturation.  This  effect  is  depicted  on  imagery  in  Figure  60. 


Figure  59:  Diagram  of  spherical  aberration  (Mansurov,  2011) 
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i 

Figure  60:  Example  of  spherical  aberration  variation.  Left:  1050  nm.  Right:  1550nm  Note  less  saturation 
at  image  center  of  1550nm.  Light  intensity  pattern  is  due  to  the  Airy  diffraction  for  the  lens  configuration. 

6.1.3  Vignette 

Vignette  affects  the  outer  portions  of  image  plane.  Vignette  is  exhibited  by  dark 
regions  near  image  comers.  Mechanical  vignette  is  a  result  of  excess  distance  between 
the  filter  and  fore  optic  or  undersized  filter  diameters  which  limit  peripheral  light  at  the 
image  plane.  Optical  vignette  is  shown  on  a  diagram  in  Figure  61,  where  central  light 
enters  at  a  larger  aperture  than  peripheral  light. 


Figure  61 :  Diagram  of  optical  vignette  (Mansurov,  2011) 
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6.2  Appendix  B:  Sensor  Preprocessing  Code 


6.2.1  flightSensorFovCalc.m 

function  [suasSd,  suasVIS,  specTir,  monoSd] =  f lightSensorFovCalc ( isKnown_wDist) 

%f light Sens orFovCalc (isKnown_wDist) 

%  INPUTS  :  isKnown_wDist=  (0)if  wDist  is  being  solved  for 

%  (1)  if  wDist  is  known  and  calculations  are  needed  for  calibration 

%  OUTPUTS  :  suasSd  =  imaging  system  parameters  for  SUAS  SWIR 

%  suasVIS  =  imaging  system  parameters  for  SUAS  VIS 

%  specTir  =  imaging  system  parameters  for  Ground  Based  HSI 

%  monoSd  =  imaging  system  parameters  from  (Peskosky, 2010) 

%  Author:  Stephen  Sweetnich  AFIT  2013 

%%  UAS  Specific  Parameters 

velBody  =  [12  0  0]  ;  %Velocity,  Body  Frame  [x,  y,  z]  (m/s) 

sensorDeg  =  30  ;  %angle  of  elevation  rel  to  yz  frame (deg) 

perlnVu  =  0.50;  %Percentage  of  horiz  FOV  to  keep 

alt  =  33;  %  altitude (AGL)  (m) 

wDist  =  alt/sind ( sensorDeg) ;  %Solve  for  wDist 

%alt  =  sind (sensorDeg) . *wDist;  %altitude (AGL)  %Solve  for  alt 

%%  NIIRS  coefficients 

c0=10 . 251 ;  cl=-3 . 16;  c2=2.817;  c3  =  -0.334;  c4  =  -0.656; 

%%  SUAS  Skin  Detection  System  parameters  (SWIR) 
suasSd=struct; 

suasSd. hSensor  =0.0160;  %m  (Goodrich  KTS1.7) 
suasSd. vSensor  =0.0128;  %m 
suasSd. hRes  =  640;  %horiz .  resolution  pxls 
suasSd. vRes  =  512;  %vert.  resolution  pxls 
suasSd. fLength  =  0.035;  %lens  focal  length  (m) 

%  For  single  lens  system  to  find  FOV 

suasSd. hFOVdeg  =  atand ( (suasSd. hSensor/2) . /suasSd . fLength) . *2 ;  %  horizontal  FOV  (deg) 

suasSd. vFOVdeg  =  atand ( (suasSd. vSensor/2) . /suasSd . fLength) . *2 ;  %  vertical  FOV  (deg) 

if  isKnown_wDist  ==  0  %  If  solving  for  working  distance 

%pixel  size  (to  solve  for  working  distance) 

suasSd. hSizePxl  =  0.05;  %Horizonal  Size  of  pixel  on  target (m) 
suasSd. vSizePxl  =  0.05;  %Vertical  Size  of  pixel  on  target (m) 

suasSd. wDist  =  suasSd. hSizePxl. /tand( suasSd. hFOVdeg/suasSd. hRes) ;  %  working  distance  (m) 
else  %  Else  wdist  is  known 
suasSd. wDist  =  60;  %m 

suasSd. hSizePxl  =  tand (suasSd. hFOVdeg/suasSd. hRes) . *suasSd . wDist; 
suasSd. vSizePxl  =  tand(suasSd.vFOVdeg/suasSd. vRes) . *suasSd . wDist; 

end 

suasSd. hFOV  =  tand ( suasSd .hFOVdeg . /2) . *suasSd ,wDist*2 ;  % (m) 

suasSd. vFOV  =  tand ( suasSd .vFOVdeg . /2) . *suasSd .wDist*2 ;  % (m) 

suasSd. pArea  =  suasSd. hFOV. *suasSd. vFOV;  %pixel  area  (mA2) 

suasSd. tPxls  =  suasSd. hRes. *suasSd. vRes;  %total  pixels 

suasSd. pDens  =  suasSd. tPxls . /suasSd. pArea;  %pixel  density  (pxl/mA2) 

suasSd. GSDh  =  2*tand (suasSd . hFOVdeg/ (2*suasSd. hRes) ) * suasSd. wDist;  %  meters 

suasSd. GSDv  =  2*tand(suasSd. vFOVdeg/ (2*suasSd. vRes) ) /cosd (90-sensorDeg) *suasSd. wDist;  %  meters 
suasSd. GSD  =  (suasSd. GSDh  *  suasSd. GSDv) A (0 . 5) ; 
suasSd. SNR  =52;  %  average  SNR  for  CCD 

suasSd. NIIRS  =  cO  +cl*logl0 (suasSd. GSDh/ . 0254 )  +c2*logl0 (0 . 9)  +c3*l/suasSd. SNR  +  c4*l.l; 

%%  SUAS  Skin  Detection  System  parameters  (VIS) 
suasVIS=struct; 

suasVIS . hSensor  =0.008195;  %m  (HHD  Omnivision) 
suasVIS .vSensor  =0.007535;  %m 
suasVIS. hRes  =  1920;  %horiz .  resolution  pxls 
suasVIS. vRes  =  1080;  %vert.  resolution  pxls 
suasVIS . fLength  =  0.012;  %lens  focal  length  (m) 

suasVIS . hFOVdeg  =  atand (( suasVIS . hSensor/2) . /suasVIS . fLength) . *2 ;  %  horizontal  FOV  (deg) 

suasVIS .vFOVdeg  =  atand ( (suasVIS. vSensor/2) . /suasVIS. fLength) . *2;  %  vertical  FOV  (deg) 

if  isKnown_wDist  ==  0  %  If  solving  for  working  distance 
%pixel  size  (to  solve  for  working  distance) 

suasVIS .hSizePxl  =  0.025;  %Horizonal  Size  of  pixel  on  target (m) 
suasVIS .vSizePxl  =  0.025;  %Vertical  Size  of  pixel  on  target (m) 
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%  For  single  lens  system  to  find  FOV 

suasVIS . wDist  =  suasVIS . hSizePxl . /tand ( suasVIS . hFOVdeg/suasVIS . hRes) ;  %  working  distance  (m) 
else  %  Else  wdist  is  known 
suasVIS .wDist  =  60;  %m 

suasVIS .hSizePxl  =  tand (suasVIS .hFOVdeg/suasVIS .hRes) . *suasVIS . wDist; 
suasVIS .vSizePxl  =  tand (suasVIS . vFOVdeg/suasVIS .vRes) . *suasVIS . wDist; 

end 

suasVIS. hFOV  =  tand (suasVIS .hFOVdeg. /2) . *suasVIS.wDist*2;  % (m) 

suasVIS. vFOV  =  tand (suasVIS .vFOVdeg . /2) . *suasVIS . wDist*2;  % (m) 

suasVIS .pArea  =  suasVIS .hFOV. *suasVIS .vFOV;  %pixel  area  (mA2) 

suasVIS . tPxls  =  suasVIS .hRes . *suasVIS .vRes;  %total  pixels 

suasVIS .pDens  =  suasVIS .tPxls . /suasVIS .pArea;  %pixel  density  (pxl/mA2) 

suasVIS. GSDh  =  2*tand (suasVIS .hFOVdeg/ (2*suasVIS. hRes) ) *suasVIS . wDist;  %  meters 

suasVIS. GSDv  =  2*tand (suasVIS .vFOVdeg/ (2*suasVIS . vRes )) /cosd ( 90-sensorDeg) *suasVIS . wDist;  %  meters 
suasVIS. SNR  =  36;  %Manufacturer  spec  sheet  =36dB 

suasVIS .NIIRS  =  cO  +cl*logl0 (suasVIS . GSDh/ . 0254)  +c2*logl0 (0 . 9)  +c3*l/ suasVIS . SNR  +  c4*l.l; 

%%  SpecTIR  HSI  Parameters 

specTir=struct; 

specTir . hSensor  =0.0160;  %m 

specTir . vSensor  =0.0128;  %m 

specTir. hRes  =  640;  %horiz.  resolution  pxls 

specTir. vRes  =  320;  %vert.  resolution  pxls 

specTir . f Length  =  0.023;  %lens  focal  length  (m) 

%  For  single  lens  system  to  find  FOV 

specTir . hFOVdeg  =  atand ( (specTir .hSensor/2) . /specTir . fLength) . *2;  %  horizontal  FOV  (deg) 

specTir .vFOVdeg  =  atand ( (specTir. vSensor/2) . /specTir. fLength) . *2;  %  vertical  FOV  (deg) 

if  isKnown_wDist  —  0 

%pixel  size  (to  solve  for  working  distance) 

specTir . hSizePxl  =  0.10;  %Horizonal  Size  of  pixel  on  target (m) 
specTir .vSizePxl  =  0.10;  %Vertical  Size  of  pixel  on  target (m) 

specTir .wDist  =  specTir . hSizePxl . /tand ( specTir . hFOVdeg/specTir . hRes) ;  %  working  distance  (m) 

else 

specTir .wDist  —  60;  %m 

specTir . hSizePxl  =  tand (specTir . hFOVdeg/specTir . hRes) . *specTir . wDist; 
specTir .vSizePxl  =  tand (specTir . vFOVdeg/specTir .vRes) . *specTir . wDist; 

end 

specTir. hFOV  =  tand (specTir .hFOVdeg. /2) . *specTir.wDist*2;  % (m) 
specTir. vFOV  =  tand (specTir .vFOVdeg . /2) . *specTir . wDist*2 ;  % (m) 
specTir .pArea  =  specTir . hFOV. *specTir .vFOV;  %pixel  area  (mA2) 
specTir . tPxls  =  specTir .hRes . *specTir .vRes;  %total  pixels 
specTir .pDens  =  specTir .tPxls . /specTir .pArea;  %pixel  density  (pxl/mA2) 

specTir. GSDh  =  2*tand (specTir .hFOVdeg/ (2*specTir. hRes) ) *specTir . wDist;  %  meters 

specTir. GSDv  =  2*tand (specTir .vFOVdeg/ (2*specTir. vRes) ) /cosd (90-sensorDeg) * specTir . wDist;  %  meters 
specTir. SNR  =  66.4;  %  Manufacturer  spec  sheet. 8 00:1  =  66.84  dB 

specTir .NIIRS  =  cO  +cl*logl0 (specTir . GSDh/ . 0254)  +c2*logl0 (0 . 9)  +c3* 1/specTir . SNR  +  c4*l.l; 

%%  Monocular  Skin  Detection  Camera  (Peskosky,  2010) 
monoSd  =  struct; 

monoSd. hSensor  =0.0160;  %m  (Goodrich  KTS1.7) 
monoSd. vSensor  =0.0128;  %m 
monoSd. hRes  =  640;  %horiz .  resolution  pxls 
monoSd. vRes  =  512;  %vert.  resolution  pxls 
monoSd. fLength  =  0.150;  %lens  focal  length  (m) 

%  For  single  lens  system  to  find  FOV 

monoSd. hFOVdeg  =  atand ( (monoSd. hSensor/2) . /monoSd . fLength) . *2 ;  %  horizontal  FOV  (deg) 

monoSd. vFOVdeg  =  atand ( (monoSd. vSensor/2) . /monoSd . fLength) . *2 ;  %  vertical  FOV  (deg) 

if  isKnown_wDist  —  0 

%pixel  size  (to  solve  for  working  distance) 

monoSd. hSizePxl  =  0.05;  %Horizonal  Size  of  pixel  on  target (m) 
monoSd. vSizePxl  =  0.05;  %Vertical  Size  of  pixel  on  target (m) 

monoSd. wDist  =  monoSd. hSizePxl. /tand (monoSd . hFOVdeg/monoSd . hRes) ;  %  working  distance  (m) 

else 

monoSd. wDist  =  6;  %m 

monoSd. hSizePxl  =  tand (monoSd. hFOVdeg/monoSd. hRes) . *monoSd . wDist; 
monoSd. vSizePxl  =  tand (monoSd. vFOVdeg/monoSd. vRes) . *monoSd . wDist; 

end 

monoSd. hFOV  =  tand (monoSd .hFOVdeg . /2) . *monoSd ,wDist*2 ;  % (m) 
monoSd. vFOV  =  tand (monoSd .vFOVdeg . /2) . *monoSd ,wDist*2 ;  % (m) 
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monoSd.pArea  =  monoSd.hFOV. *monoSd. vFOV;  %pixel  area  (mA2) 
monoSd.tPxls  =  monoSd.hRes. *monoSd. vRes;  %total  pixels 
monoSd.pDens  =  monoSd.tPxls . /monoSd.pArea;  %pixel  density  (pxl/mA2) 
monoSd.GSDh  =  2*tand(monoSd.hF0Vdeg/ (2*monoSd.hRes) ) *monoSd. wDist;  %  meters 

monoSd.GSDv  =  2*tand (monoSd . vFOVdeg/ (2*monoSd. vRes) ) /cosd ( 90-sensorDeg) *monoSd. wDist;  %  meters 


6. 2. 2  ND  GRI video  Viewer,  six 


Figure  62:  Block  Diagram  for  Simulink  NDGR1  video  viewer 


6.2.3  NDSIVideoViewer.slx 


B 


Figure  63:  Block  Diagram  for  Simulink  NDSI  video  viewer 
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6.3  Appendix  C:  Waypoint  Visualization  Code 


6.3.1  tlog2flat.m 


function  fltData  =  tlog2f lat (inputTlogtxt, homeLat, homeLong,  outputFileName) 
%function  tlog2flat (inputTlogtxt, outputFileName) 

% 

%  Convert  dataflash  data  to  6DOF  Flat  Earth  model 


%  INPUT: 

%  inputTlogtxt :  tlog  text  file  that  is  generated  in  APM  tlog 

%  extractor.  See  Tlog 

%  data  extractor.  Format  for  config  file: 

%  mavlink_gps_raw_int_t  time_usec 

%  mavlink_global_position_int_t  lat 

%  mavlink_global_position_int_t  Ion 

%  mavlink_global_position_int_t  alt 

%  mavlink_attitude_t  roll 

%  mavlink_attitude_t  pitch 

%  mavlink_attitude_t  yaw 

%  homeLat:  home  latitude  near  airfield 

%  homeLong:  home  latitude  near  airfield 

%  %Himsel  Field 

%  %homeLat  =  39.343233;  %  degrees 

%  %homeLong  =  -86.029917;  %  degrees 

%  %SUAS  Airfield 

%  %39. 34422;  %  degrees 

%  %-86. 00939;  %  degrees 


outputFileName  (optional) :  filename  string  you  wish  to  save  data  as 


%  OUTPUT:  flat  earth  coordinate  system  6DOF  model 
%  n  x  7  double  array 

%  [time,  X,  Y,Z,  roll, pitch, yaw, timeComputer] 

%  Author:  Stephen  Sweetnich  AFIT  2013 


%%  Create  flight  data  structure 

%file  =inputTlogtxt ;  %.csv  file  From  Tlog  Data  Extractor 
%  homeLat  =  39.343233;  %  degrees 

%  homeLong  =  -86.029917;  %  degrees 

file  =  'I:\SUASFlightTest\19Novl3_FlightTest\tlogs\Extracted_2013-ll-19  1 l-40-58_Flat3 . csv ' 
%file  =  ' f It2data210ctl3_2 . csv' ;  %.csv  file  From  Tlog  Data  Extractor 
fid  =  fopen(file); 

rawData  =  textscan ( f id,  ' %s  %s  %f  %f  %f  %f  %f  %f  %f'  ,  ' delimiter headerlines ', 1 ') ; 
f close ( ' all ' ) ; 


%  GPS  home  reference  (Update  for  every  flight  if  home  airfield  changes) 
if  nargin  —  1 

%  Specify  starting  locaiton  of  GPS 
for  i  =  1 : length (rawData { 1, 3} ) 

if  rawData{ 1, 3 } (i)  ~=0  &&  rawData { 1 , 4 } (i)  ~=0 

homeLat  =  rawData {1, 4 } (i) ; 
homeLong  =  rawData { 1, 5} (i) ; 
break 

end 

end 


end 


time  =  rawData{ 1, 3 } . /1000000;  %time 
lat  =  rawData{l, 4 } . /10000000; %  degrees 
long  =  rawData{ 1, 5 } . /10000000 ;  %  degrees 

alt  =  rawData {1, 6} . /I 000; 

%  Intitialize  time  at  GPS  syncronization  start 
x  —  0 ;  i  =  1; 
while  x  ~=  1 

if  time(i)  ~=  0  &&  lat(i)~=0  &&  long(i)~=0  &&  alt(i)  ~=0 
x  —  1 ; 

else 

i  =  i+1; 

end 

end 
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timeComputer  =  rawData{ 1, 2 } (i : end-100 ) /  %  time  of  computer 

href  =  180;  %meters  MSL  for  atterbury  field 

roll  =  rad2deg (rawData{ 1, 7 } (i :end-100) ) ;  %  deg 

pitch  =  rad2deg (rawData { 1 , 8 } (i : end-100) ) ;  %  deg 

yaw  =  rad2deg (rawData{l, 9} (i:end-100) ) ;  %  deg 

time2  =  linspace (0, time (end) -time (i) , length (yaw) )' ; 


%ecefCoordXYZ  =*  lla2ecef ([ lat (i : eol) , long ( i : eol) , alt (i : eol) ]) ; 
flatCoordXYZ  =  lla2f lat ( [lat (i :end-100) , long (i :end-100) , -alt (i :end-100) ] , . . . 
[homeLat, homeLong] , 0, href) ; 

%  fltData  =  struct; 

%  fltData. time  =  time2; 

%  fltData. X  =  ecefCoordXYZ (:, 1 ) ; 

%  fltData. Y  =  ecefCoordXYZ (:, 2) ; 

%  fltData. Z  =  ecefCoordXYZ (:, 3) ; 

%  fltData. phi  =  roll; 

%  fltData . theta  -  pitch; 

%  fltData. psi  =  yaw; 

fltData  =  { time2, flatCoordXYZ ( : , 1 ) , flatCoordXYZ ( : ,2) , flatCoordXYZ ( : ,  3) , . . . 

roll  , pitch,  yaw, timeComputer (i : end-100 )} ; 
if  nargin  —  4 

save (outputFileName, ' fltData' ) ; 

%  save ( 'flt2Data210ctl3_flat.mat' , ' fltData' ) ; 

End 


6.3.2  df2flat _plusIMU.m 


function  fltData  =  df2f lat_plusIMU (inputDataFlashMat, homeLat,  homeLong,  outputFileName) 
%function  df2flat (inputDataFlashMat, outputFileName) 


%  Convert  dataflash  data  to  6DOF  Flat  Earth  model  and  adds  high  rate  IMU  data 

% 

%  INPUT: 

%  inputDataFlashMat:  Dataflash  .mat  file  that  is  generated  in  APM 

%  outputFileName (optional)  :  filename  string  you  wish  to  save  data  as 

%  homeLat:  home  latitude  near  airfield 

%  homeLong:  home  latitude  near  airfield 

%  %  Himsel  Field 

%  %homeLat  =  39.343233;  %  degrees 

%  %homeLong  =  -86.029917;  %  degrees 

%  %  SUAS  Airfield 

%  %homeLat  =  39.34422;  %  degrees 

%  %homeLong  =  -86.00939;  %  degrees 


%  OUTPUT:  flat  earth  coordinate  system  6DOF  model 
%  n  x  7  double  array 

%  [time,  X, Y, Z,  roll, pitch, yaw,  GyrX, GyrY, GyrZ, AccX, AccY, AccZ] 


%  Author:  Stephen  Sweetnich  AFIT  2013 


%  load  flight  data  structure 

S  =  load  (inputDataFlashMat);  % '2013-10-21  16-26  6. log. mat' 

%S  =  load (  'I:\SUASFlightTest\21Octl3_FlightData\dataflashLogs\2013-10-21_16-29_7.1og.mat') 
%%  Specify  logging  rates  from  Autopilot 

%  (User  Selects  proper  logging  rates) 
gpsOf f set  =  S . GPS (1,1) ; 

gpsRate  =  5;  %Hz  For  APM  MediaTek  GPS 
gpsLines  =  S . GPS (end, 1) ; 

attRate  —  10;  %50;  %Hz  For  APM  2.6  DataFlash 
attLines  =  S . ATT (end, 1) ; 

imuRate  =  50;  %10;  %Hz  For  APM  2.6  DataFlash 
imuLines  —  S . IMU (end, 1) ; 

%%  Select  parameter  with  highest  rate 

%  Interpolations  are  made  between  the  highest  rate  data 
[highestRate,  param]  =  max ( [attRate, imuRate] ) ; 
if  param  ==1 ; 

totLines  =S . ATT (end, 1) ; 

else 
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totLines  =S . IMU (end, 1) ; 


end 


%%  Initialize  data  structure  assembly 

%  Preallocate  data  structure 

t  =  zeros (totLines, 1) ; 

dt  =  zeros (totLines, 1) ; 

lat  =  zeros (totLines, 1) ; 

long  =  zeros (totLines, 1 ) ; 

alt  =  zeros (totLines, 1) ; 

roll  =  zeros (totLines , 1 ) ; 

pitch  =  zeros (totLines, 1) ; 

yaw  =  zeros (totLines, 1) ; 

gyrX  =  zeros (totLines, 1) ; 

gyrY  =  zeros (totLines , 1 ) ; 

gyrZ  =  zeros (totLines, 1) ; 

accX  =  zeros (totLines, 1 ) ; 

accY  =  zeros (totLines, 1) ; 

accZ  =  zeros (totLines,!); 


%  Run  a  loop  on  structure  and  fill  in  values  for  rawData 
x=0 ; i=l ; 

for  i  =1: totLines 

if  i  <  size (S. GPS, 1) 

ml  =  S.GPS (i, 1) -1; 
t (ml)  =  S.GPS (i, 3) / 1 0  0  0 ; 
lat (ml)  =  S.GPS  (i, 6) ; 
long (ml)  =  S.GPS(i,7); 
alt (ml)  =  S.GPS (i,8) ; 

end 

if  i  <  size (S. ATT, 1) 
m2  =  S. ATT (i, 1) ; 
roll (m2 ) =  S .ATT (i, 2 ) ; 
pitch (m2)  =  S.ATT(i,3); 
yaw (m2)  =  S .ATT  (i, 4) ; 

end 


if  i  <  size (S . IMU, 1) 
m3  =  S. IMU (i, 1) ; 


end 


gyrX (m3) 
gyrY (m3) 
gyrZ (m3) 
accX (m3) 
accY (m3) 
accZ (m3) 


S . IMU ( 
S . IMU ( 
S . IMU ( 
S . IMU ( 
S . IMU ( 
S . IMU ( 


t  (i-1) =t (i) ; 
lat (i-1) =lat (i)  ; 
long(i-l)=  long(i); 
alt (i-1) =alt (i) ; 


end 


end 

%  Populate  raw  data  structure  with  all  non  zero  values  if  possible 
x=0 ; k=l ; c=l ; 
while  x  ~=1 

if  (t (k) ~=0  ||  lat (k) ~=0  ||  long(k)~=0  ||  alt (k) ||  ... 

roll(k)~=0  ||  pitch (k)~=0  ||  yaw(k)  ~=0  ||  ... 

gyrX (k)  ~=0 | | gyrY (k)  ~=0  ||  gyrZ(k)  ~=0) 


rawData 

(0,1) 

=  t(k)  ; 

rawData 

(c,2) 

=  lat (k) ; 

rawData 

(c,3) 

=  long (k) ; 

rawData 

(c,  4 ) 

-  alt (k) ; 

rawData 

(0,5) 

=  roll (k) ; 

rawData 

(c,6) 

=  pitch (k) 

rawData 

(o,7) 

-  yaw (k) ; 

rawData 

(c,8) 

=  gyrX(k); 

rawData 

(o,9) 

=  gyrY (k) ; 

rawData 

(c, 10) 

=  gyrz(k) 

rawData 

(0,11) 

-  accX(k) 

rawData 

(c,12) 

-  accY(k) 

rawData 
c=c+l ; 

(o,13) 

-  accZ(k) 

end 
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end 


if  k  —  length (t) 
x=l  ; 

end 

k=k+l; 


%%  Linearly  interpolate  between  GPS  readings 

%  If  GPS  time  is  same  as  next  reading,  make  it  zero 
for  1  =  2 : length (rawData) 

if  rawData (1, 1) ==rawData (1-1, 1) 
rawData (1,1)  =  0; 
rawData (1,2)  =  0; 
rawData (1,3)  =  0; 
rawData (1,4)  =  0; 

end 

end 


idl=l ;  id2=2 ; 

for  i=l : length ( rawData) 

if  [rawData (i, 1 : 4 ) ]  ~=  [0  0  0  0] 
id2  =i; 

delta  =  id2-idl; 
for  j=l : (delta-1) 
for  k=l : 4 

%  Interpolation  for  time 

rawData (id2- j , k) =  ( ( (delta- j ) /delta) * (rawData (id2 , k) -rawData ( idl, k) ) ) +rawData (idl , k) ; 

%  %  Interpolation  for  lat 

%  rawData (id2-j , 2 ) =  ( ( (delta- j ) / delta) * (rawData (id2 , 2 ) -rawData ( idl, 2) ) ) +rawData (idl , 2 ) 

%  %  Interpolation  for  long 

%  rawData (id2- j , 3) =  ( ( (delta- j ) /delta) * (rawData (id2 , 3) -rawData ( idl, 3) ) ) +rawData (idl , 3) 

%  %  Interpolation  for  alt 

%  rawData (id2- j , 4) =  ( ( (delta- j ) / delta) * (rawData (id2 , 4 ) -rawData ( idl, 4) ) ) +rawData (idl , 4 ) 

end 

end 

idl=id2 ; 

else 

id2=i ; 

end 

end 

%%  Linearly  interpolate  between  ATT 

cols  =  [5:7]  ; 

idl=l;  id2=2; 

for  i=l : length (rawData) 

if  [rawData (i, cols) ]  ~=  zeros (1 , length (cols ) ) 
id2  =i; 

delta  =  id2-idl; 
for  j=l : (delta-1) 

for  k=  1 : length (cols) 

%  Interpolation  for  time 

rawData (id2-j , cols (k) ) =  ( ( (delta- j ) /delta) * (rawData (id2, cols (k) ) - 
rawData (idl,  cols (k) ) ) ) frawData (idl, cols (k) ) ; 
end 

end 

idl=id2 ; 

else 

id2=i ; 

end 

end 

%%  Linearly  interpolate  between  IMU  readings 

cols  =[8:13]; 

idl=l;  id2=2; 

for  i=l : length (rawData) 

if  [rawData (i, cols) ]  ~=  zeros (1 , length (cols ) ) 
id2  =i; 

delta  =  id2-idl; 
for  j=l : (delta-1) 

for  k=  1 : length (cols) 

%  Interpolation  for  time 

rawData (id2-j , cols (k) ) =  ( ( (delta- j ) /delta) * (rawData (id2, cols (k) ) - 
rawData (idl, cols (k) ) ) ) frawData (idl,  cols (k) ) ; 
end 

end 

idl=id2 ; 

else 

id2=i; 

end 

end 
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%%  Clip  bad  data  from  beginning  when  GPS  hasn't  initiailized 

%  find  when  GPS  was  synchronized  and  when  it  was  lost 

nonz  =  find (S .GPS ( : , 6) ) ; 

gdata  =  sort(nonz); 

clipLine  =  S . GPS (gdata ( 1) , 1 ) ; 

lastLine  =  S . GPS (gdata (end) , 1 ) ; 

epochTime  =S . GPS (gdata (1) , 3) /1000; 

% 

%  for  i=l : length (S . GPS) ; 

%  if  S.GPS (i, 3) ~=0 

%  break 

%  else 

%  i=i+l; 

%  end 

%  end 

%  epochTime  =  S .GPS (i, 3) /1000; 

%  clipLine  =  S.GPS(i,l); 

%%  build  structure  in  11a  system 
totLines  =  size (rawData, 1 ) -clipLine ; 

%  Preallocate  data  structure 
llaData  =  zeros (totLines, 7); 


c=clipLine; 
for  i=l: totLines 
llaData (i,l)  = 
llaData (i, 2)  = 
llaData (i, 3)  = 
llaData (i, 4)  = 
llaData (i, 5)  = 
llaData (i, 6)  = 
llaData (i, 7)  = 
llaData (i, 8)  = 
llaData (i, 9)  = 
llaData (i, 10)  = 
llaData (i, 11)  = 
llaData (i, 12)  = 
llaData (i, 13)  = 
c=c+l ; 

end 

%%  Convert  to  flat 


rawData  (c,  1)  -e; 
rawData (c, 2) ; 
rawData (c, 3) ; 
rawData (c, 4) ; 
rawData (c, 5) ; 
rawData (c, 6)  ; 
rawData (c, 7) ; 
rawData (c, 8) ; 
rawData (c, 9) ; 

:  rawData (c, 10) 

:  rawData (c, 11) 

;  rawData (c, 12) 

:  rawData (c, 13) 


pochTime;  %time(s) 
%lat  degrees 
%long  degrees 
%alt (m) 

%roll  rad 
%pitch  rad 
%yaw  rad 

%imu  GyrX 
%imu  GyrY 
;  %imu  GyrZ 
;  %imu  AccX 
;  %imu  AccY 
;  %imu  AccZ 


earth  model  system 


%  GPS  home  reference  (Update  for  every  flight  if  home  airfield  changes) at 
if  nargin  —  1 

%  Specify  starting  locaiton  of  GPS 

homeLat  =  llaData (1,2) ;  % (deg)  Use  GPS  data  after  initialized 
homeLong  =  llaData ( 1, 3) ;  %(deg) 

end 


%  Approximate  Location  of  skin  dismounts 
%  d_Lat  -  homeLat; 

%  d_Long  =  homeLong; 

%Altitude  MSL  for  atterbury  field 
href  =-180;  %-216;  % (m) 

%ecef CoordXYZ  =  lla2ecef (llaData ( : , 2 : 4 ) ; 
flatCoordXYZ  =  lla2f lat (llaData ( : , 2 : 4) , . . . 

[homeLat, homeLong] , 0,  href ,  'WGS84'  ); 

%  fltData  =  struct; 

%  fltData. time  =  time2; 

%  fltData. X  =  ecefCoordXYZ ( : , 1 ) ; 

%  fltData. Y  =  ecefCoordXYZ (:, 2 ) ; 

%  fltData. Z  =  ecefCoordXYZ (:, 3) ; 

%  fltData. phi  =  roll; 

%  fltData . theta  =  pitch; 

%  fltData. psi  =  yaw; 

fltData  =  [llaData ( : , 1) , flatCoordXYZ ( : , 1) , flatCoordXYZ ( : , 2) , abs (flatCoordXYZ ( : , 3) ) , . . . 

llaData  (: ,5:13) ] ; 
if  nargin  ==4 

save (outputFileName, ' fltData ' )  ; 
end 
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6.4  Appendix  D:  Sensor  Postprocessing  Code 


6.4.1  GenerateROCs 


%  Generate  ROC  Curves  from  ENVI  files  (specifically  for  Ground  HSI  imager) 

%  Utilizes  roc()  and  auroc()  by  Fawcett,  T.,  "ROC  graphs  :  Notes  and  practical 
%  considerations  for  researchers".  Technical  report,  HP 

%  Laboratories,  MS  1143,  1501  Page  Mill  Road,  Palo  Alto 

%  CA  94304,  USA,  April  2004. 

%  Input  the  ENVI  files  with  gui  and  generate  ROC  curves 

%  Author:  Stephen  Sweetnich  AFIT  2013 

clear  all 

clc 

%%  Get  Image  Data 

%  ENVI  Import  Cube  and  Register  images 

[datafile,  datapath,  ~]  =  uigetfile (['*.*'],' Select  a  data  file'); 

%hdrfile  —  [datafile (1 : end-4)  '.hdr']; 
hdrfile  =  [datafile  '.hdr']; 

[truth_file,  truthpath,~]  =  uigetf ile ('*. txt ',' Select  a  truth  file'); 

[I,  hdr]  =  enviread ( [datapath  datafile] , [datapath  hdrfile]); 

%nl  =  permute (I,  [213]);  %  Changes  Image  to  [row  x  lines  x  bands] 

%% 

nl  =  I  ;%  permute  not  needed  for  processed  ENVI  data 
img  =  imrotate (nl , 90,  'nearest'); 

%clear  nl  I 

sizeDetection  =  size (img); 


%%  Use  ENVI  Detection 
%  %  Apply  Bandmath 
%  %new  imager 
%  %1080nm:  band  139 
%  %1580nm :  band  219 
%  %660nm:  band  58 
%  %540nm:  band  33 

%  For  4  band  file,  comment  out  this: 

%  ndsi  =  (img ( : , : , 139) -img ( : , : , 219) ) . / (img ( : , : , 139) +img ( : , : , 219) ) ;  Calculate  NDSI 

%  ndgri  =  (img ( : ,  : , 58) -img  ( : ,  : , 33) )  . / (img  ( : , : , 58) +img ( : , : , 33) ) ;  Calculate  NDGRI 

%For4bandf ile 

ndsi  =  (img ( : , : , 3) -img ( : , : , 4) )  . / (img ( : ,  : , 3) +img ( : ,  : , 4) )  ; 
ndgri  =  (img ( : , : , 2) -img ( : , : , 1) ) . / (img ( : , : , 2) +img ( : , : , 1) ) ; 

ndgriThreshold=0 . 025; 

ndsiROC  =  [ ] ; 

for  i=l : size (ndsi, 1) 

for  j=l : size (ndsi, 2 ) 

if  ndgri (i,j)  >  ndgriThreshold 
ndsiROC (i,j)  =  ndsi(i,j); 

else 

ndsiROC (i,j)  =  ndsi(i,j)-l; 

end 

end 

end 

%  Normalize  the  data  and  plot  result 
%f igure (1) 

normalizeNDSI  =  @ (x)  (x  +2)  .*  (1  -  0)  ./  (1  +  2)  +  0; 

detectNorm  =  normalizeNDSI (ndsiROC) ; 
imshow ( detectNorm) 

%  detectNorm  =  mapminmax (img ( : , : , 1) , 0, 1) ; 

%%  Import  Truth  file 

%  Select  Option  in  ENVI  to  Output  ROI  to  ASCII 
fid  =  f open ( [truthpath  truth_file] ) ; 
format  =  '%u  %u  %u  %u ' ; 

ROI  =  textscan ( f id,  format,  'Headerlines',  8); 

truthIM  =zeros (sizeDetection ([2,1])); 
for  i=l :ROI [1, 1 } (end) 

truthIM (ROI {1,3} (i) , ROI {1,2} (i) )=1; 

end 
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truthIM  =  imrotate ( truthIM,  90 ) ; 
%figure(2);  axis  image 
%imshow (truthIM) ; 


%%  Reshape  data 
sizeIM=  size (detectNorm) ; 

target  =  reshape (truthIM, 1,  sizelM (1) *sizeIM (2) ) ; 
output  =  reshape (detectNorm, 1 , sizelM ( 1) *sizeIM (2) ) ; 

%%  Generate  ROC  Curve 
figure (3) ; 
hold  on 

[tp,  fp]  =  roc2 (target ' ,  output'); 
auROC  =  auroc (tp, fp) ; 
plot (fp, tp, 'b ' ) ; 

%% 

legend (labels) 

xlabel (' False  Positive  Rate ' , ' FontSize ' ,  16); 
ylabel('True  Positive  Rate FontSize ' ,  16); 

%  title (' SpecTIR  Receiver  Operating  Charachteristic  (ROC)'); 
%  text (0 . 5, 0 . 5, [ ' Area  Under  Curve  =  '  num2str (auROC) ] ) ; 

%hold  off 


6.5  Appendix  E:  Relevant  TRB/SRB  Documentation 

6.5.1  Test  Objective  1  -  Verification  of  Autopilot  Gain  Settings  and 
Configuration 


TEST  SCENARIO  1 

Description 

Determine  appropriate  gain  settings  for  autonomous  operation. 

Stakeholders 

Stephen  R.  Sweetnich,  AFIT  /ENV 

Success 

Criteria 

Test  Matrix 

Test  Point 

Description 

Altitude  AGL 
(±3  Oft) 

Radius 

No  Patterns  | 

1.1 

Gain  Test 

164  ft 

N/A 

N/A 

NOTE:  Approximate  gain  settings  will  be  entered  before  gain  tuning.  Gain 
settings  for  throttle  will  be  speed  controlled  at  approximately  30  mph  (26  kts) 

Evaluation 

Criteria 

Satisfactory  if: 

Working  gain  settings  are  achieved  for  normal  flight. 

Data 

Requirements 

Required 

1 .  Approximate  gain  settings. 

2.  Telemetry  and  mission  planner  software. 
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Algorithms 

Mission  Planner  Tuning  interface 

Expected 

Results 

1 .  Tuned  gain  settings  for  APM 

2.  Verification  of  configuration  hardware  and  software. 

Assets 

AFIT  Electric  Sig  Rascal  with  APM  2.5  and  2.74  FW 

Test 

Methodology 

Test  Procedures 

1.  BEFORE  TAKEOFF: 

a.  Setup  ground  control  station  and  operating  area  1AW  AFIT  Document 
5028. 

b.  Complete  all  required  preflight  checklists  for  UAS. 

c.  Check  that  weather  is  within  limits  and  determine  launch/recovery 
locations  and  headings. 

d.  Open  airspace  with  range  control. 

2.  LAUNCH: 

a.  Ensure  that  all  present  personnel  are  aware  of  launch. 

b.  Position  aircraft  for  ground  launch. 

c.  Safety  pilot  executes  takeoff. 

d.  Safety  pilot  announces  that  aircraft  is  airborne. 

e.  Climb  to  pre-briefed  transition  altitude. 

f.  Transition  to  pre-briefed  test-point  entry  position. 

3.  EXECUTE  TEST  POINTS: 

a.  Ensure  GS  is  recording  telemetry  file  for  entire  flight. 

b.  Initiate  flight  with  stabilize  mode. 

c.  Ensure  gain  settings  are  appropriate 

d.  Write  test  waypoints,  enter  autopilot  mode  and  ensure  gain  stability 
when  crossing  from  ‘stabilize’  to  ‘auto’. 

e.  Once  patterns  finished,  execute  recovery 

4.  RECOVERY: 

a.  Navigate  aircraft  to  pre-briefed  recovery  transition  location. 

b.  Transition  aircraft  to  safety  pilot  control  or  observation. 

c.  Ensure  landing  area  is  clear  of  personnel  and  equipment. 

d.  Begin  descent  and  entry  into  landing  pattern. 

e.  Safety  pilot  announces  landing  to  all  present  personnel. 

f.  Execute  recovery. 

5.  AFTER  RECOVERY: 

a.  Stop  telemetry  capture  on  laptop  or  ensure  that  data  log  is  saved. 

b.  Close  airspace  with  range  control. 
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c.  Disconnect  aircraft  battery  power  prior  to  moving  aircraft  by  hand. 

d.  Power  off  RC  transmitter  as  required. 

e.  Power  off  video  capture  equipment  as  required. 

f.  Power  off  ground  control  station  as  required. 


6.5.2  Test  Objective  2-  Capture  Multispectral  Video  Data 


TEST  SCENARIO  2 


Description 


Collect  high  resolution  linear  and  rotational  acceleration  data  from  autopilot 
onboard  Electric  Sig  Rascal. 


Stakeholders 


Stephen  R.  Sweetnich,  AFIT  /ENV 


Success 

Criteria 


Test  Matrix 


Test 

Point 

Description 

Sensor 

Angle 

Filter 

Rate 

Altitude 

AGL 

(±10ft) 

No 

Patterns 

2.1 

Racetrack  1 

30° 

High 

186  ft 

3 

2.2 

Racetrack  2 

30° 

High 

140  ft 

3 

2.3 

Racetrack  3 

30° 

High 

100  ft 

3 

2.4 

Racetrack  4 

30° 

Low 

186  ft 

3 

2.5 

Racetrack  5 

30° 

Low 

140  ft 

3 

2.6 

Racetrack  6 

30° 

Low 

100  ft 

3 

2.7 

Racetrack  7 

45° 

Low 

186  ft 

3 

2.8 

Racetrack  8 

45° 

Low 

140  ft 

3 

2.9 

Racetrack  9 

45° 

Low 

100  ft 

3 

2.10 

Racetrack  10 

45° 

High 

186  ft 

3 

2.11 

Racetrack  11 

45° 

High 

140  ft 

3 

2.12 

Racetrack  12 

45° 

High 

100  ft 

3 

Note:  Throttle  setting  from  gain  tuning  (slow  as  possible)  will  be  used  and 
_ maintained.  Throttle  will  be  speed  controlled  at  approximately  30  mph  (26  kts) 

Evaluation  Satisfactory  if: 

Criteria  Imagery  data  is  collected  for  both  sensors.  Data  quality  is  adequate  for  video 

processing. 


Data 


Required 
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Requirements 

1.  Tuned  gain  settings  for  settings  for  ArduPlane  2.74  settings.  See 

Objective  1. 

2.  Telemetry  and  mission  planner  software. 

3 .  Video  systems  are  able  to  collect  footage. 

Algorithms 

1 .  Multispectral  imagery  via  Goodrich  S WIR  and  mini  DVR 

2.  VIS  EO  imagery  via  HackHD 

3 .  Software  to  actuate  multiple  rotating  disks  via  filter  exchange  unit 

Expected 

Results 

High  resolution  (50Hz)  accelerometer  and  gyro  data  is  collected  with  APM's  on¬ 
board  16MB  datalogging  memory  and  GS  via  Mission  Planner. 

Assets 

1.  AFIT  Electric  Sig  Rascal  with  APM  2.5  and  2.74  FW 

Test 

Methodology 

Test  Procedures 

1.  BEFORE  TAKEOFF: 

a.  Setup  ground  control  station  and  operating  area  1AW  AFIT  Document 
5028. 

b.  Preload  UAS  waypoint  profile  with  designated  flight  paths. 

c.  Ensure  APM  telemetry  data  link  with  Mission  Planner  and  GCS  logging 
is  enabled. 

d.  Complete  all  required  preflight  checklists  for  UAS. 

e.  Check  that  weather  is  within  limits  and  determine  launch/recovery 
locations  and  headings. 

f.  Open  airspace  with  range  control. 

2.  LAUNCH: 

a.  Ensure  that  all  present  personnel  are  aware  of  launch. 

b.  Position  aircraft  for  ground  launch. 

c.  Safety  pilot  executes  takeoff. 

d.  Safety  pilot  announces  that  aircraft  is  airborne. 

e.  Climb  to  pre-briefed  transition  altitude. 

f.  Transition  to  pre-briefed  test-point  entry  position. 

3.  EXECUTE  TEST  POINTS: 

a.  Ensure  GS  is  recording  telemetry  file  for  entire  flight. 

b.  Initiate  flight  with  stabilize  mode. 

c.  Ensure  gain  settings  are  appropriate  (Objective  1)  before  switching  to 
auto  mode. 

d.  Fly  appropriate  racetrack  at  each  desired  condition  in  the  test  matrix 
(test  points  2.1-2.12). 

e.  Write  waypoints,  enter  autopilot  mode 

f.  Once  patterns  finished,  execute  recovery 
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4.  RECOVERY: 

a.  Navigate  aircraft  to  pre-briefed  recovery  transition  location. 

b.  Transition  aircraft  to  safety  pilot  control  or  observation. 

c.  Ensure  landing  area  is  clear  of  personnel  and  equipment. 

d.  Begin  descent  and  entry  into  landing  pattern. 

e.  Safety  pilot  announces  landing  to  all  present  personnel. 

f.  Execute  recovery. 

5.  AFTER  RECOVERY: 

a.  Stop  telemetry  capture  on  laptop  or  ensure  that  data  log  is  saved. 

b.  Close  airspace  with  range  control. 

c.  Disconnect  aircraft  battery  power  prior  to  moving  aircraft  by  hand. 

d.  Power  off  RC  transmitter  as  required. 

e.  Power  off  video  capture  equipment  as  required. 

f.  Power  off  ground  control  station  as  required. 


6.5.3  Test  Objective  3-  Measure  High  Resolution  INS  Data 


TEST  SCENARIO  3 


Description 


Stakeholders 


Success 

Criteria 


Collect  high  resolution  linear  and  rotational  acceleration  data  from  autopilot 
onboard  Electric  Sig  Rascal. 


Stephen  R.  Sweetnich,  AFIT  /ENV 


Test  Matrix 


Test  Point 

Description 

Altitude  AGL 
(±10  ft) 

Radius/Standoff 

No  Patterns 

3.1 

Racetrack  1 

186  ft 

82  ft 

3 

3.2 

Racetrack  2 

140  ft 

82  ft 

3 

3.3 

Racetrack  3 

100  ft 

82  ft 

3 

Note:  Throttle  setting  from  gain  tuning  will  be  used  and  maintained.  Throttle 
_ will  be  speed  controlled  at  approximately  30  mph  (26  kts) _ 

Evaluation  Satisfactory  if: 

Criteria  INS  data  at  is  collected  from  one  source,  for  all  test  patterns. 


Data 

Requirements 


Required 

1.  Tuned  gain  settings  for  settings  for  ArduPlane  2.74  settings.  See 
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Objective  1. 

2.  Telemetry  and  mission  planner  software. 

3 .  High  collection  rate  for  onboard  Autopilot 

Algorithms 

1 .  Data  Recorded  from  Ardupilot: 

a.  Accelerometer  X,  Y,Z  axes  (mg) 

b.  Gyroscope  Pitch,  Roll,  Yaw  rates. (rad/s) 

c.  Speed  (m/s) 

Expected 

Results 

High  resolution  (50Hz)  accelerometer  and  gyro  data  is  collected  with  APM's  on¬ 
board  16MB  datalogging  memory  and  GS  via  Mission  Planner.  Also,  a  backup 
INS  will  log  information. 

Assets 

AFIT  Electric  Sig  Rascal  with  APM  2.5  and  2.74  FW 

Test 

Methodology 

Test  Procedures 

1.  BEFORE  TAKEOFF: 

a.  Setup  ground  control  station  and  operating  area  IAW  AFIT  Document 
5028. 

b.  Preload  UAS  waypoint  profile  with  designated  flight  paths. 

c.  Ensure  APM  telemetry  data  link  with  Mission  Planner  and  GCS  logging 
is  enabled. 

d.  Complete  all  required  preflight  checklists  for  UAS. 

e.  Check  that  weather  is  within  limits  and  determine  launch/recovery 
locations  and  headings. 

f.  Open  airspace  with  range  control. 

2.  LAUNCH: 

a.  Ensure  that  all  present  personnel  are  aware  of  launch. 

b.  Position  aircraft  for  ground  launch. 

c.  Safety  pilot  executes  takeoff. 

d.  Safety  pilot  announces  that  aircraft  is  airborne. 

e.  Climb  to  pre-briefed  transition  altitude. 

f.  Transition  to  pre-briefed  test-point  entry  position. 

3.  EXECUTE  TEST  POINTS: 

a.  Ensure  GS  is  recording  telemetry  file  for  entire  flight. 

b.  Initiate  flight  with  stabilize  mode. 

c.  Ensure  gain  settings  are  appropriate  before  switching  to  auto  mode. 

d.  Execute  racetracks  for  test  points  3. 1-3. 3. 

e.  Once  patterns  finished,  execute  recovery 

4.  RECOVERY: 

a.  Navigate  aircraft  to  pre-briefed  recovery  transition  location. 
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b.  Transition  aircraft  to  safety  pilot  control  or  observation. 

c.  Ensure  landing  area  is  clear  of  personnel  and  equipment. 

d.  Begin  descent  and  entry  into  landing  pattern. 

e.  Safety  pilot  announces  landing  to  all  present  personnel. 

f.  Execute  recovery. 

5.  AFTER  RECOVERY: 

a.  Stop  telemetry  capture  on  laptop  or  ensure  that  data  log  is  saved. 

b.  Close  airspace  with  range  control. 

c.  Disconnect  aircraft  battery  power  prior  to  moving  aircraft  by  hand. 

d.  Power  off  RC  transmitter  as  required. 

e.  Power  off  OSD  video  capture  equipment  as  required. 

f.  Power  off  ground  control  station  as  required. 
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